How to Optimize Agent Cost and Latency—Without Breaking Behavior

Overview

Take a support-triage agent: it reads an incoming ticket, decides whether it needs a knowledge-base lookup, drafts a reply, and files the ticket under a category. On day one it runs comfortably against a shortlist of test tickets. Six weeks and forty thousand tickets later, it's timing out under load and burning through its model budget well before the month is out. The fix that goes in under pressure is a model swap or a truncated prompt: latency drops, and a few days later a support lead is asking why closed tickets are getting reopened.

Cost and latency are easy to watch. Behavior isn't. A dashboard shows token count and p95 latency in real time; nobody has a dashboard for "quietly worse reasoning," and by the time it surfaces as a support escalation the connection to a model swap two weeks back isn't obvious to anyone looking. That asymmetry is why teams over-optimize the number they can see and under-notice the one they can't.

The patterns below (model mixing, lazy evaluation, caching, context trimming, and tool-output compression) do the same thing from different angles. Each one moves cheap, reversible work off the expensive model and onto something cheaper, without touching the step where precision actually matters. None of them is a trick. They are closer to bookkeeping.

Every code sample below uses the OpenAI Python SDK, but the pattern is provider-agnostic: it maps cleanly onto Anthropic, an open-weight model served locally, or any mix of the three. Treat the model names as placeholders for whatever you actually run.

Model Mixing: Fast Models to Think, Slow Models to Act

The highest-leverage change on this list is splitting planning and execution across two models: a fast, cheap model for planning, idea generation, and candidate actions, and a slow, expensive model reserved for final decisions, tool calls, and anything irreversible.

We think this is the change worth making first, and the reason is structural rather than about any particular model. Planning is forgiving: a slightly clumsy plan usually gets caught and corrected downstream. Execution is not; a single wrong tool call can write to a database or send an email that can't be unsent. A compiler runs cheap heuristic passes before committing to its expensive optimization pass for exactly this reason. Agents benefit from the same shape.

A small model is reliably cheaper per token than a frontier one, often by a wide margin, which means moving the high-volume, low-stakes planning chatter onto it is usually the single largest cost reduction you can make without touching the agent's actual answers.

from openai import OpenAI
client = OpenAI()

def fast_plan(question):
    res = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {"role": "system", "content": "Draft a high-level plan. No tool calls."},
            {"role": "user", "content": question}
        ]
    )
    return res.choices[0].message.content

def slow_execute(plan):
    res = client.chat.completions.create(
        model="gpt-4.1",
        messages=[
            {"role": "system", "content": "Using the plan, execute the task with high precision."},
            {"role": "user", "content": plan}
        ]
    )
    return res.choices[0].message.content

plan = fast_plan("Summarize these documents and find contradictions.")
result = slow_execute(plan)

Model mixing breaks in a few predictable ways: the fast model hallucinates a plan step that doesn't hold up, the slow model interprets the plan too literally, or the two models were given inconsistent personas and start working against each other.

The literal-interpretation failure is the sneakiest of the three. A frontier executor faithfully carries out whatever the plan says, mistakes included, because it's been told to trust the plan. If the fast planner invents a step, the slow model won't second-guess it. It will execute it perfectly. The fix isn't a smarter executor. It's a gate between the two stages.

def validate_plan(plan):
    res = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {"role": "system", "content": "Is the plan complete, safe, and executable? Respond true/false."},
            {"role": "user", "content": plan}
        ]
    )
    return "true" in res.choices[0].message.content.lower()

Model mixing doesn't have to be all-or-nothing. The more robust setups route per step rather than per request, escalating to the slow model only when a step crosses some complexity or risk threshold:

• read-only and reversible steps stay on the fast model
• writes, payments, deletions, and anything user-facing escalate to the slow model
• steps the fast model flags as low-confidence escalate automatically

That keeps the frontier model on the critical path only where its precision earns its cost, and it gives you one dial, the escalation threshold, to trade accuracy against spend without rewriting the agent.

Lazy Evaluation: Skip the Step You Don't Need

Plenty of agents evaluate every retrieved document, every tool call, every clarification, every prior message in the conversation, even when most of that work was never necessary in the first place.

Lazy evaluation, borrowed from functional programming, just means deferring work until its result is actually demanded. We think a surprising amount of what an agent does is never demanded at all. A question the model already knows the answer to doesn't need retrieval. A request that's already unambiguous doesn't need a clarifying turn. Each skipped step saves a round-trip, and round-trips dominate latency far more than token count does.

def maybe_retrieve(question):
    needs_rag_prompt = f"""
Do we need external documents to answer this: "{question}"?
Respond yes or no.
"""
    res = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": needs_rag_prompt}]
    )
    if "yes" in res.choices[0].message.content.lower():
        return rag_search(question)  # expensive fetch
    return []

The gating call here is cheap; the retrieval it guards is not. Skipping a vector search, and the tokens those documents would have added to every later prompt in the loop, is a compounding win. The context you didn't retrieve doesn't ride along on every subsequent turn either.

The same gate works before a tool call ("is a tool call required for this plan, yes or no") and before a clarifying question ("does this request already have enough information to act on"). Both cost one cheap call and remove most of the unnecessary ones while still letting the genuinely ambiguous cases through. An unnecessary clarifying question is worse than it looks, too: it costs the user a full extra turn, and it costs your pipeline a full extra round-trip on top of it.

The failure mode is the same for every gate above: the cheap model says "no" when the honest answer was "yes," and the agent confidently produces an answer it could have looked up. Lazy evaluation should fail toward doing the work, not skipping it. When the gating model is uncertain, default to retrieving, calling the tool, or asking the question. A false skip breaks the answer, which is exactly the regression these patterns are supposed to avoid.

Caching: Don't Pay for the Same Reasoning Twice

Most agent tasks repeat reasoning. The same document gets summarized again and again across requests. The same schema gets re-derived. The same context gets re-analyzed. Each repetition is a full inference you already paid for once, and caching the intermediate result turns a recurring cost into a one-time one.

import hashlib

reasoning_cache = {}

def cache_key(prompt):
    return hashlib.sha256(prompt.encode()).hexdigest()

def cached_reason(prompt, model="gpt-4o-mini"):
    key = cache_key(prompt)
    if key in reasoning_cache:
        return reasoning_cache[key]
    res = client.chat.completions.create(
        model=model,
        messages=[{"role": "user", "content": prompt}]
    )
    answer = res.choices[0].message.content
    reasoning_cache[key] = answer
    return answer

The in-memory dict here is the simplest possible store; in production you'd swap it for Redis or any shared cache so the savings survive process restarts. The SHA-256 key requires an exact prompt match, which cuts both ways: it never returns a stale answer for a different prompt, but it also misses prompts that are semantically identical and differ only by a stray whitespace character. For near-duplicates, key off an embedding-similarity bucket instead of the raw string.

This is an application-level cache: you store and reuse the model's output yourself. It's distinct from, and complementary to, the native prompt caching the major providers now offer at the API layer, which reuses internal computation over a repeated prompt prefix rather than the final answer. The two work best together. Native caching rewards you for keeping large, static content (system prompt, tool definitions, few-shot examples) at the front of the prompt and variable, user-specific content at the end, so the stable prefix stays cacheable across calls. Application caching, meanwhile, skips the call entirely for reasoning you've already produced. Use the provider's cache to make each call cheaper, and your own cache to make fewer calls.

Context Window: Send Less, Not Everything

Large prompts mean higher latency, higher cost, and higher hallucination risk, and the last of those matters most for behavior. Beyond a certain size, more context doesn't make an agent smarter. It makes it more distractible. Irrelevant passages compete for the model's attention, dilute the tokens that actually matter, and tend to push the most important facts toward the middle of a long prompt, which is roughly where models attend to them least. Trimming context is a cost lever and an accuracy lever at once.

The recurring patterns are relevance filtering, semantic dedupe, chunk scoring, summarization, and swapping a raw transcript for a key-value memory.

def get_relevant_chunks(query, chunks, top_k=5):
    scored = [(chunk, similarity(query, chunk)) for chunk in chunks]
    return [c for c, _ in sorted(scored, key=lambda x: x[1], reverse=True)[:top_k]]

Here similarity is typically cosine similarity between embedding vectors. Tune top_k empirically: too low and you starve the model of context it needed, too high and you reintroduce the bloat you set out to remove. Five chunks is a reasonable starting point to adjust from against an evaluation set.

def compress_context(context):
    res = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {"role": "system", "content": "Summarize in 5 bullet points. No details, only facts."},
            {"role": "user", "content": context}
        ]
    )
    return res.choices[0].message.content

Compression trades a cheap extra call for a much smaller payload on the expensive one. That's usually a clear win, since the fast model summarizes once while the slow model would otherwise re-read the full text on every loop.

The caveat is lossiness: a summary that drops the one figure or clause the task hinges on will quietly produce a wrong answer. Keep verbatim anything a downstream step must quote or compute on, and compress only the narrative around it.

Replaying an entire transcript on every turn is the most common source of slow, expensive agents, because the prompt grows without bound as the conversation continues. Swapping in the last one or two user messages, the agent's last reply, a rolling memory summary, and the task plan is almost always enough, and it keeps the prompt's stable prefix intact, so native prompt caching keeps paying off too.

Compressing Tools and Results

If your tools return huge JSON blobs, full logs, raw markdown, complete HTML, or verbose error messages, you're paying for tokens you don't need on every loop. Tool output is uniquely expensive because it lands back in the prompt and rides along on every subsequent step. A single bloated API response doesn't cost you once. It costs you on every turn until it scrolls out of context, which makes tool results the highest-leverage place to compress.

def compress_tool_result(raw):
    res = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {"role": "system", "content": "Compress tool output to essential data. No fluff."},
            {"role": "user", "content": raw}
        ]
    )
    return res.choices[0].message.content

Before reaching for an LLM, check whether plain code can do the job. Most of the worst offenders (pretty- printed JSON, HTML wrappers, stack-trace noise, repeated boilerplate) can be stripped deterministically with a parser or a few lines of filtering, at zero added latency and with no risk of a summarizer dropping a field. Save the LLM compressor for genuinely unstructured output where rules fall short.

def compress_json(raw_json):
    prompt = f"""
Given this JSON, output a minimal JSON keeping only:
- IDs
- titles
- status
- essential fields
Compress, remove null or unused fields.
JSON:
{raw_json}
"""
    return cached_reason(prompt)

Routing this through cached_reason means an identical tool response only ever gets compressed once. The cache and the compressor reinforce each other. For anything mission-critical, validate the compressed JSON against a schema before trusting it, so a malformed summary fails loudly instead of silently corrupting the next step.

Putting the Pipeline Together

Each pattern helps on its own, but they compound when composed. Fast planning shrinks the work; lazy evaluation removes work entirely; compression shrinks what survives; caching makes sure nothing gets paid for twice.

skip retrieval --
|yes
Retrieve + compress on fast model v -----------------------------------------> Reasoning cached? |hit -> reuse cached result --
|miss -> compute + cache it
v Write, payment, or irreversible step? |no -> handle on fast model --
|yes -> escalate to slow model
v Response -->

Figure 1 — The routing pipeline. Every gate is a chance to spend nothing at all; the slow model only runs at the end, and only when the step in front of it actually needs it.

def optimized_agent(question):
    plan = fast_plan(question)
    docs = maybe_retrieve(question)
    compressed_docs = [compress_context(d) for d in docs]
    final = slow_execute(
        f"Plan:\n{plan}\n\nCompressed documents:\n{compressed_docs}"
    )
    return final

Notice what the slow, expensive model actually sees: a compact plan and a handful of compressed summaries, never the raw question-to-answer firehose. The frontier call runs on the smallest, most relevant prompt the rest of the pipeline could produce, which is precisely where its cost is justified.

We'd expect a pipeline like this to bring cost and latency down substantially while holding accuracy roughly flat, and in several cases trimming context bloat makes the agent more stable rather than less. Treat that as a direction, not a guarantee. The actual numbers depend on your task mix, how repetitive your reasoning is, and how bloated your starting prompts were, and the only way to know is to measure your own pipeline rather than borrow someone else's figure.

The discipline that makes any of this safe is measurement: hold an evaluation set fixed and watch accuracy as you turn each pattern on, one at a time. If a change moves cost down and accuracy stays flat, keep it. If accuracy slips, you've found the boundary: back off the threshold rather than abandoning the pattern.

Conclusion

None of the five patterns here is exotic. Route by model, skip work nothing needs, reuse what you've already reasoned through, send less context, and compress what tools hand back. What makes them an engineering discipline rather than a grab-bag of tricks is the constraint tying them together: every one moves cheap, reversible work off the expensive model and never touches the step where a mistake is hard to undo.

The order to tackle them in is roughly the order they appear above. Model mixing pays for itself fastest because it changes nothing about what the agent sees, only which model reads it. Everything after that is about giving both models less to do in the first place.