How to Optimize Agent Cost and Latency—Without Breaking Behavior

Overview

Take a support-triage agent: it reads a ticket, decides whether it needs a knowledge-base lookup, drafts a reply, and files the ticket under a category. On day one it runs fine on a shortlist of test tickets. Six weeks and forty thousand tickets later, it's timing out under load and burning its model budget before month end. The fix that goes in under pressure is a model swap or a truncated prompt: latency drops, and a few days later a support lead is asking why closed tickets are getting reopened.

Cost and latency are easy to watch. Behavior isn't. A dashboard shows token count and p95 latency in real time; nobody has one for "quietly worse reasoning," and by the time it surfaces as an escalation, the connection to a model swap two weeks back is invisible.

We care about which lever does what, because two of the patterns below are widely sold as latency fixes when they are not. Every price and limit quoted here was read from the provider's own documentation on 29 September 2026, and every one has moved at least once in the past year.

Two Different Problems With Different Fixes

Almost every "make the agent faster and cheaper" thread collapses cost and latency into one number. They behave nothing alike. Cost is close to linear in tokens, whether they arrived in one request or twelve. Latency is dominated by two things a token counter barely shows: how many tokens the model has to generate, and how many times your process waits for a round trip.

OpenAI's latency-optimization guide states the asymmetry plainly. Generating tokens "is almost always the highest latency step when using an LLM: as a general heuristic, cutting 50% of your output tokens may cut ~50% of your latency." On the input side, the same guide says "cutting 50% of your prompt may only result in a 1–5% latency improvement." Anthropic separates the same two measurements, baseline latency and time to first token, and its one structural recommendation for perceived speed is streaming, which changes no tokens at all.

Table 1 — Eight levers, and the fact that only three move both numbers the same way.

The last row settles the argument. Anthropic now sells latency as its own SKU: fast mode for Claude Opus 5.5 is $8 per million input tokens and $40 per million output, against $4 and $20 at standard speed. Speed is not a free side effect of the patterns below, or nobody would charge a 2x premium for it.

Figure 1 — The routing question, asked per step rather than per request. Two of the three branches cost you nothing in accuracy.

Model Mixing: Fast Models to Think, Slow Models to Act

The highest-leverage change here is splitting planning and execution across two models: a fast, cheap model for planning and candidate actions, and a slow, expensive one reserved for final decisions, tool calls, and anything irreversible.

We think this is the change worth making first, and the reason is structural. Planning is forgiving: a clumsy plan usually gets caught downstream. Execution is not; one wrong tool call can write to a database or send an email that can't be unsent.

The spread is wide enough to make the split worth the plumbing. Claude Haiku 4.5 is $1 per million input tokens and $5 per million output, against $5 and $25 for Claude Opus 5 and $10 and $50 for Claude Fable 5.1. On OpenAI's pricing page, gpt-5-nano is $0.05 per million input tokens against $10 for gpt-6-astra, a factor of 200.

from openai import OpenAI
client = OpenAI()

def fast_plan(question):
    res = client.chat.completions.create(
        model="gpt-5-nano",
        messages=[
            {"role": "system", "content": "Draft a high-level plan. No tool calls."},
            {"role": "user", "content": question}
        ]
    )
    return res.choices[0].message.content

def slow_execute(plan):
    res = client.chat.completions.create(
        model="gpt-6-astra",
        messages=[
            {"role": "system", "content": "Using the plan, execute the task with high precision."},
            {"role": "user", "content": plan}
        ]
    )
    return res.choices[0].message.content

Model mixing breaks in three predictable ways: the fast model hallucinates a plan step, the slow model reads the plan too literally, or the two were given inconsistent instructions. The fix isn't a smarter executor. It's a gate between the two stages.

def validate_plan(plan):
    res = client.chat.completions.create(
        model="gpt-5-nano",
        messages=[
            {"role": "system", "content": "Is the plan complete, safe, and executable? Respond true/false."},
            {"role": "user", "content": plan}
        ]
    )
    return "true" in res.choices[0].message.content.lower()

Route per step rather than per request: read-only steps stay on the fast model, while writes, payments, deletions and anything flagged low-confidence escalate.

Lazy Evaluation: Skip the Step You Don't Need

Plenty of agents evaluate every retrieved document, every tool call, every clarification and every prior message, even when most of that work was never necessary.

Lazy evaluation, borrowed from functional programming, just means deferring work until its result is actually demanded. We think a surprising amount of what an agent does is never demanded at all. A question the model already knows the answer to doesn't need retrieval. A request that's already unambiguous doesn't need a clarifying turn. Each skipped step removes a round trip, and "make fewer requests" sits at number four on OpenAI's list of seven latency principles for exactly that reason.

def maybe_retrieve(question):
    needs_rag_prompt = f"""
Do we need external documents to answer this: "{question}"?
Respond yes or no.
"""
    res = client.chat.completions.create(
        model="gpt-5-nano",
        messages=[{"role": "user", "content": needs_rag_prompt}]
    )
    if "yes" in res.choices[0].message.content.lower():
        return rag_search(question)  # expensive fetch
    return []

The gating call is cheap; the retrieval it guards is not. Skipping a vector search also skips the tokens those documents would have added to every later prompt. The same gate works before a tool call and before a clarifying question, and an unnecessary clarifying question costs the user a full extra turn and your pipeline a full extra round trip.

The failure mode is the same for every gate: the cheap model says "no" when the honest answer was "yes," and the agent confidently produces an answer it could have looked up. Lazy evaluation should fail toward doing the work. When the gating model is uncertain, retrieve, call the tool, or ask.

Caching: Don't Pay for the Same Reasoning Twice

Two different things are called caching here, they have opposite failure modes, and confusing them is how teams leave money on the table.

An application-level cache stores the model's output and skips the call. It is the one you write.

import hashlib

reasoning_cache = {}

def cache_key(prompt):
    return hashlib.sha256(prompt.encode()).hexdigest()

def cached_reason(prompt, model="gpt-5-nano"):
    key = cache_key(prompt)
    if key in reasoning_cache:
        return reasoning_cache[key]
    res = client.chat.completions.create(
        model=model,
        messages=[{"role": "user", "content": prompt}]
    )
    answer = res.choices[0].message.content
    reasoning_cache[key] = answer
    return answer

In production you'd swap the dict for Redis so the savings survive restarts. The SHA-256 key needs an exact match, so it never returns a stale answer but misses prompts that differ by one whitespace character. For near-duplicates, key off an embedding-similarity bucket.

Provider prompt caching works on prefixes rather than whole prompts. Anthropic charges 1.25x the base input price to write a 5-minute entry and 0.1x to read it, so break-even is one read; the 1-hour variant costs 2x and needs two. Minimum cacheable length runs from 512 tokens on Claude Opus 5.5 and Sonnet 5.5 up to 4,096 on Claude Haiku 4.5, you get at most four explicit breakpoints, and the entry "is refreshed for no additional cost each time the cached content is used." OpenAI's caching is on by default, needs 1,024 tokens on GPT-5.6 and later, prices cached input at 0.1x, and keeps an entry for "30 minutes after its most recent write or reuse."

The prefix rule is where this goes wrong. OpenAI lists what resets a prefix: a change of model, of tool definitions, of output schema, of reasoning effort, or context compaction. So opening a system prompt with the current timestamp, or the user's ID, silently converts every request into a full-price miss. Static content first, volatile content last.

Context Window: Send Less, Not Everything

Trimming context is a cost and accuracy lever, barely a latency one: half the prompt buys a few percent of the wall clock.

The accuracy argument is the better one anyway. Liu and colleagues, in "Lost in the Middle", found that "performance is often highest when relevant information occurs at the beginning or end of the input context, and significantly degrades when models must access relevant information in the middle of long contexts, even for explicitly long-context models." Anthropic's compaction documentation says the same from the vendor side: it keeps the active context small "because response quality degrades as a conversation grows." Past a certain point more context doesn't make an agent smarter. It makes it more distractible.

The patterns worth having are relevance filtering, semantic dedupe, summarization, and swapping a raw transcript for a key-value memory.

def get_relevant_chunks(query, chunks, top_k=5):
    scored = [(chunk, similarity(query, chunk)) for chunk in chunks]
    return [c for c, _ in sorted(scored, key=lambda x: x[1], reverse=True)[:top_k]]

Tune top_k empirically: too low starves the model of context it needed, too high reintroduces the bloat you set out to remove.

def compress_context(context):
    res = client.chat.completions.create(
        model="gpt-5-nano",
        messages=[
            {"role": "system", "content": "Summarize in 5 bullet points. No details, only facts."},
            {"role": "user", "content": context}
        ]
    )
    return res.choices[0].message.content

Compression trades a cheap extra call for a much smaller payload on the expensive one. The caveat is lossiness: keep verbatim anything a downstream step must quote or compute on, and compress only the narrative around it.

There is a second, less obvious cost. Anthropic's context-editing documentation is explicit that clearing tool results "invalidates cached prompt prefixes when content is cleared," which is why the feature has a clear_at_least parameter: clear enough tokens to be worth the cache write you just triggered. Trimming and caching pull against each other, and a trim that fires every turn can cost more than the context it removes.

Compressing Tools and Results

Tool output is uniquely expensive because it lands back in the prompt and rides along on every subsequent step: a single bloated API response costs you on every turn until it scrolls out of context.

Anthropic's pricing page puts a typical 10 kB web page at about 2,500 tokens, a large documentation page at about 25,000, and a 500 kB research PDF at about 125,000 tokens. A single unfiltered PDF fetch can therefore consume more input budget than a thousand ordinary turns. Tool definitions are not free either: enabling tools at all adds a system prompt of 286 to 675 tokens depending on the model, the computer-use toolset adds roughly 4,500 input tokens, and the browser toolset roughly 6,600, on every single request.

def compress_tool_result(raw):
    res = client.chat.completions.create(
        model="gpt-5-nano",
        messages=[
            {"role": "system", "content": "Compress tool output to essential data. No fluff."},
            {"role": "user", "content": raw}
        ]
    )
    return res.choices[0].message.content

Before reaching for an LLM, check whether plain code can do the job. Pretty-printed JSON, HTML wrappers, stack-trace noise and repeated boilerplate all strip deterministically with a parser, at zero added latency and no risk of a summarizer dropping a field.

def compress_json(raw_json):
    prompt = f"""
Given this JSON, output a minimal JSON keeping only:
- IDs
- titles
- status
- essential fields
Compress, remove null or unused fields.
JSON:
{raw_json}
"""
    return cached_reason(prompt)

Routing this through cached_reason means an identical tool response only ever gets compressed once. Validate the compressed JSON against a schema so a malformed summary fails loudly instead of corrupting the next step.

The cheapest version is to prune the tool list: an agent carrying forty tool definitions it never calls pays for all forty on every turn, and you control the effective list at the authorization layer, not in the prompt.

Buying and Selling Latency

The lever nobody reaches for first is the one with the best price: if no human is waiting, stop paying for synchronous inference.

Both providers discount asynchronous work by half. Anthropic's Message Batches API charges "50% of the standard API prices" on input, output and special tokens, accepts up to 100,000 requests or 256 MB per batch, and finishes most batches "in less than 1 hour" inside a 24-hour window, after which unfinished requests expire unbilled. OpenAI's Batch API gives the same "50% cost discount compared to synchronous APIs" with "each batch completes within 24 hours (and often more quickly)", returning a JSONL file whose "output line order may not match the input line order." Its flex processing prices ordinary requests at batch rates in exchange for "slower response times and occasional resource unavailability", and wants the request timeout raised from its 10-minute default to about 15.

Anthropic's batch discount stacks with caching: a nightly re-categorization job over a stable system prompt pays half of 0.1x on the cached portion. Backfills, eval runs, bulk classification and overnight summaries all belong here.

Putting the Pipeline Together

Each pattern helps on its own, but they compound when composed.

skip retrieval --
|yes
Retrieve + compress on fast model v -----------------------------------------> Reasoning cached? |hit -> reuse cached result --
|miss -> compute + cache it
v Write, payment, or irreversible step? |no -> handle on fast model --
|yes -> escalate to slow model
v Response -->

Figure 2 — The composed pipeline. Every gate is a chance to spend nothing at all; the slow model runs at the end, and only when the step in front of it needs it.

def optimized_agent(question):
    plan = fast_plan(question)
    docs = maybe_retrieve(question)
    compressed_docs = [compress_context(d) for d in docs]
    final = slow_execute(
        f"Plan:\n{plan}\n\nCompressed documents:\n{compressed_docs}"
    )
    return final

Notice what the slow model actually sees: a compact plan and a handful of compressed summaries, never the raw firehose. That is precisely where its cost is justified.

The Arithmetic on Forty Thousand Tickets

Percentages are easy to argue with, so here is the triage agent from the overview priced at published rates. Assume 40,000 tickets a month, 6,000 input tokens per ticket once the system prompt, tool definitions, ticket body and retrieved article are counted, and 400 output tokens.

Table 2 — The same agent under three configurations, at the rates published on 29 September 2026.

The cache line is worth unpacking. Of the 6,000 input tokens on the triage call, 4,000 are the system prompt and tool definitions, identical every time. At 40,000 tickets a month a read lands roughly every minute, well inside the 5-minute lifetime, and each read refreshes the entry for free, so the write cost rounds to nothing. Those 160 million tokens bill at $0.10 per million instead of $1.00.

That is a 54% reduction, and the model that writes the customer-facing reply never changed. None of it tells you whether accuracy held. Hold an evaluation set fixed, turn on one pattern at a time, and watch accuracy while cost moves. If accuracy slips, you've found the boundary; back off the threshold rather than abandoning the pattern.

Frequently Asked Questions

Does trimming my prompt make the agent faster? Barely. OpenAI's latency guide puts a 50% prompt reduction at "a 1–5% latency improvement". Trim input to cut cost and to keep the model focused; cut output tokens and round trips to cut latency.

How much does prompt caching actually save? A cache read is 0.1x the base input price on both providers. Anthropic's 5-minute write costs 1.25x, so the entry pays for itself on the first read; the 1-hour write costs 2x and needs two. Below the per-model minimum, currently 512 to 4,096 tokens on Claude models and 1,024 on GPT-5.6 and later, nothing is cached at all.

Why did my cache hit rate collapse after a deploy? Almost certainly a prefix change. OpenAI documents that a new model, changed tool definitions, a changed output schema, a different reasoning effort, or compaction all reset the prefix. A timestamp or user ID at the top of the system prompt does the same thing on every request.

Is the Batch API worth the operational complexity? If the work is not interactive, it is a 50% discount for a scheduling change, and on Anthropic's side it stacks with caching. The constraints are the 24-hour expiry, the 100,000-request or 256 MB batch ceiling, and output ordering that "may not match the input line order."

Will a cheaper model always cost less? Not after a tokenizer change. Anthropic notes that Claude 4.7 and later models produce "approximately 30% more tokens for the same text", so a same-price upgrade can raise the bill by about a third. Measure tokens, not just rates.

Where should I start? Split planning from execution, then move the static half of your prompt to the front and let the provider cache it. Neither needs new infrastructure, and neither risks the step that matters.

Conclusion

What makes this a discipline rather than a grab-bag: every one of these patterns moves cheap, reversible work off the expensive model and never touches the step where a mistake is hard to undo.

Model mixing pays for itself fastest, and prompt caching is a close second because the provider does the work and the only cost is being disciplined about what sits at the front of your prompt. And stop treating context trimming as a speed fix: it is a good cost and accuracy fix, but the seconds live in output tokens and round trips.

Sources

All URLs read 29 September 2026. Every price and limit quoted above is from these pages and all of them change.

• Latency optimization, OpenAI — the seven principles, the ~50% output-token heuristic, and the 1–5% figure for prompt reduction.
• Prompt caching, OpenAI — caching on by default, the 1,024-token minimum for GPT-5.6 and later, 0.1x cached input, 30-minute retention, and the list of changes that reset a prefix.
• Batch API, OpenAI — the 50% discount, the 24-hour completion window, and output ordering.
• Flex processing, OpenAI — batch-rate pricing for ordinary requests, the latency trade-off, and the 10-minute default timeout.
• API pricing, OpenAI — gpt-6-astra at $10/$50 per MTok against gpt-5-nano at $0.05/$0.40.
• Pricing, Claude Developer Platform — the per-model table, cache multipliers, batch rates, fast-mode premium, tool-use system prompt token counts, per-tool overheads, and the ~30% tokenizer change on Claude 4.7 and later.
• Prompt caching, Claude Developer Platform — 1.25x and 2x writes, 0.1x reads, per-model minimum cacheable length, the four-breakpoint limit, and free refresh on use.
• Batch processing, Claude Developer Platform — 50% on all usage, most batches under an hour, the 24-hour expiry, and the 100,000-request or 256 MB ceiling.
• Reducing latency, Claude Developer Platform — baseline latency against time to first token, and streaming as the structural fix for perceived speed.
• Context editing, Claude Developer Platform — the 100,000-token default trigger, keeping three tool uses, and the cache invalidation that clear_at_least exists to offset.
• Compaction overview, Claude Developer Platform — server-side summarization, and "response quality degrades as a conversation grows".
• Lost in the Middle: How Language Models Use Long Contexts (Liu et al., 2023) — the position effect that makes context trimming an accuracy lever and not only a cost one.