Optimizing Agent Cost & Latency in Practice (LangChain, LangGraph, AutoGen Versions)

Overview

Splitting an agent into a fast planner and a slow executor is a decision you make once, on paper. Wiring it into a codebase is a decision you make three times, differently, depending on whether you are holding a LangChain agent, a LangGraph state machine, or a team of conversational agents. The pattern is identical. The code is not, and the gap is where the engineering time goes.

Naive agents do the expensive thing every time. The full conversation and every retrieved document go to the most capable model on every turn, whether the turn warrants it or not. A planning step that enumerates three subtasks pays flagship prices. A greeting triggers a vector search and a context-stuffed generation. Most of the fix is doing the cheap thing first and the expensive thing only when it earns its keep.

What has changed since this pattern was first worth writing down is that the frameworks now ship most of it. Hand-rolled caches, retry counters and call budgets have library equivalents, and the hand-rolled versions are usually worse. Versions checked on 29 September 2026: langchain 1.4.3, langgraph 1.2.12, ag2 1.1.1, autogen-agentchat 0.7.5, agent-framework 1.19.0.

Where the Money Actually Goes

Before any routing, look at the published rates, because the spread is larger than most architecture decisions. These are list prices per million tokens as published on 29 September 2026.

Three things fall out of that table. Output tokens cost four to five times input tokens on every row, so verbosity is the most expensive habit an agent can have. Cached input is an order of magnitude cheaper than fresh input, and on Anthropic's newest models more than that. And the cheapest usable model on a provider is not ten times cheaper than the flagship, it is roughly a hundred times cheaper on input, which is why the planner-on-a-small-model move is worth making even when the planner is doing real work.

Two costs are easy to forget because no dashboard breaks them out. Tool definitions are input tokens: on Claude models the tool-use system prompt alone runs from 286 tokens on Opus 5.5 to 675 on Opus 4.7, before any of your own schemas. Ship fifteen tools to an agent that needs three and you pay for the twelve on every single turn. And tokenizers are not stable across model generations. Anthropic's docs note that Claude 4.7 and later models "use a newer tokenizer" that "produces approximately 30% more tokens for the same text." A model upgrade that keeps the per-token price flat can still raise your bill.

hard-coded path -> yes: plan on small model -> retrieval needed? -- no --> execute on large model (cached prefix) -> yes: retrieve, then compress on small model -> execute on large model -> stream output, keep it short The first branch is the cheapest one and the one teams skip. -->

Figure 1 — The order the levers are worth pulling in. The top branch, not calling a model at all, is the only one that saves 100% and the only one nobody puts in a framework.

Cache Break-Even

Provider prompt caching is the lever where the arithmetic is published and unambiguous, so it is worth doing the sums rather than guessing.

On Anthropic's models, a 5-minute cache write costs 1.25x the base input price and a read costs 0.1x. The docs draw the conclusion for you: caching "pays off after one cache read for the 5-minute duration (1.25x write), or after two cache reads for the 1-hour duration (2x write)." So a stable system prompt reused twice inside five minutes is already cheaper cached. Minimum cacheable prefix length varies by model, from 512 tokens on the newest through 1,024 and 2,048 up to 4,096 on some, and you get up to 4 cache breakpoints with a 20-block lookback.

On OpenAI's side, caching applies from "1,024 visible input tokens" on GPT-5.6 and later, cached input bills at "0.1x uncached input", and for those models an entry stays available "30 minutes after its most recent write or reuse" rather than the five to ten minutes older models got. The catch is what counts as a hit: "the entire rendered prefix" must match, including "OpenAI-provided instructions, developer messages, tool definitions, and conversation history." A timestamp or a shuffled tool list at the front of your prompt silently costs you every hit.

The batch line is worth a second look because it is the largest single discount available and the most often ignored. Both providers publish a 50% discount on input and output for asynchronous batch processing, and on Anthropic's side that multiplier stacks with caching. Any agent workload that is really a nightly job pretending to be a chat interface should be on it.

LangChain: Middleware Instead of Plumbing

LangChain 1.x moved most of this out of your code. An agent is create_agent, and the cost levers attach as middleware rather than as wrappers you maintain.

from langchain.agents import create_agent
from langchain.agents.middleware import (
    SummarizationMiddleware,
    ContextEditingMiddleware,
    ModelCallLimitMiddleware,
    LLMToolSelectorMiddleware,
    ModelFallbackMiddleware,
)

agent = create_agent(
    model="anthropic:claude-sonnet-5",
    tools=ALL_TOOLS,
    middleware=[
        LLMToolSelectorMiddleware(model="anthropic:claude-haiku-4-5"),
        SummarizationMiddleware(model="anthropic:claude-haiku-4-5"),
        ContextEditingMiddleware(),
        ModelCallLimitMiddleware(thread_limit=12),
        ModelFallbackMiddleware("openai:gpt-6-luna"),
    ],
)

Each of those does a job the hand-rolled version did worse. LLMToolSelectorMiddleware uses "an LLM to select relevant tools before calling main model", which is the fix for the tool-definition overhead above. SummarizationMiddleware summarizes "conversation history when approaching token limits". ContextEditingMiddleware trims or clears tool results. ModelCallLimitMiddleware exists explicitly to "limit the number of model calls to prevent excessive costs", and ToolCallLimitMiddleware does the same for tools. ModelFallbackMiddleware fails over to an alternative model.

Caching also changed. LangChain 1.x documents provider-level caching rather than a global exact-match cache: implicit caching where "providers automatically pass on cost savings if a request hits a cache, with no configuration required", explicit control through Anthropic's cache_control or OpenAI's prompt_cache_key, and AnthropicPromptCachingMiddleware and BedrockPromptCachingMiddleware for the middleware route. Cache usage shows up in the response's usage metadata, with cache_read under input_token_details.

The planner-executor split still belongs in your code, because it is a decision about your task rather than about the library. Keep it explicit and keep the models named in one place.

from langchain.chat_models import init_chat_model

fast = init_chat_model("anthropic:claude-haiku-4-5", temperature=0)
slow = init_chat_model("anthropic:claude-sonnet-5", temperature=0)

def maybe_retrieve(question: str, retriever):
    verdict = fast.invoke(
        f"Does answering this need external documents? Answer yes or no.\n{question}"
    ).content
    return retriever.invoke(question) if "yes" in verdict.lower() else []

def compress(docs) -> str:
    if not docs:
        return ""
    joined = "\n\n".join(d.page_content for d in docs)
    return fast.invoke(f"Summarize in at most 5 bullets:\n\n{joined}").content

That yes/no call is one of the better trades available. It costs a handful of tokens on the cheap tier and it removes an embedding query plus a context-stuffed generation on every turn where the honest answer is no.

LangGraph: Cache the Node, Bound the Loop

LangGraph is worth the state object once you need branching, retries and clarification paths. Two of its features are directly cost features, and both replace code teams used to write by hand.

Node-level caching first. Instead of memoizing a function with lru_cache, attach a CachePolicy to the node and give the compiled graph a cache:

from langgraph.graph import StateGraph, START, END
from langgraph.types import CachePolicy
from langgraph.cache.memory import InMemoryCache

workflow = StateGraph(AgentState)
workflow.add_node("plan", planner_node, cache_policy=CachePolicy(ttl=300))
workflow.add_node("context", context_node, cache_policy=CachePolicy(ttl=60))
workflow.add_node("execute", execution_node)

graph = workflow.compile(cache=InMemoryCache())

The default key_func is "a hash of the input with pickle", and a cached result comes back carrying '__metadata__': {'cached': True}, which is what you assert on in a test. ttl is in seconds and entries never expire if you leave it out, which is almost never what you want for a planner whose inputs include a document that changes.

Second, the retry loop. The router below is cheap; the executor above it is not.

def router_node(state: AgentState) -> str:
    if state.get("attempts", 0) >= 3:
        return "give_up"
    conf = state.get("confidence") or 0.0
    if conf > 0.8:
        return "done"
    return "clarify" if conf > 0.5 else "retry"

workflow.add_conditional_edges(
    "execute", router_node,
    {"done": END, "clarify": "clarify", "retry": "plan", "give_up": "give_up"},
)

Figure 2 — The expensive node runs exactly once per pass. The attempts counter, not the recursion limit, is what bounds what a low-confidence question can cost you.

AutoGen, and What Came After It

The multi-agent version of this pattern is a cheap planner agent, an expensive executor agent, and a cheap critic. Which library you write it in is the part that changed.

Microsoft's autogen Python packages last shipped python-v0.7.5 on 30 September 2025, and the repository README now carries a maintenance-mode banner: AutoGen "will not receive new features or enhancements and is community managed going forward", with new users directed to Microsoft Agent Framework. The community AG2 fork kept the from autogen import ... import path and is at 1.1.1, shipping regularly. Microsoft Agent Framework itself is at 1.19.0.

For cost work, one difference in the migration guide matters more than the syntax. AutoGen's AssistantAgent "is single-turn unless you increase max_tool_iterations", while Agent Framework's Agent "is multi-turn by default and keeps invoking tools until it can return a final answer." Port a three-agent pipeline without noticing that and each agent may start looping on tools, which is a spend change dressed up as an import change.

import asyncio, json
from autogen_agentchat.agents import AssistantAgent
from autogen_ext.models.openai import OpenAIChatCompletionClient

async def run(question: str, context: str) -> dict:
    planner = AssistantAgent(
        "planner", model_client=OpenAIChatCompletionClient(model="gpt-5-nano"),
        system_message="Break the task into steps. No final answer.")
    executor = AssistantAgent(
        "executor", model_client=OpenAIChatCompletionClient(model="gpt-6-astra"),
        system_message='Follow the plan. Reply with JSON: {"answer": "..."}')
    critic = AssistantAgent(
        "critic", model_client=OpenAIChatCompletionClient(model="gpt-5-nano"),
        system_message='Reply with JSON: {"confidence": 0-1}')

    plan = (await planner.run(task=question)).messages[-1].content
    answer = json.loads((await executor.run(
        task=f"Q: {question}\nPlan: {plan}\nContext: {context}")).messages[-1].content)
    graded = json.loads((await critic.run(
        task=f"Q: {question}\nA: {answer['answer']}")).messages[-1].content)
    return {**answer, **graded}

Two of the three agents run on the cheap tier. Calling them in a fixed order rather than handing them to a free-flowing group chat is deliberate: a group chat's token usage grows with the conversation, and an explicit call order is both cheaper and predictable.

Latency Is Not Cost

This is the part that most cost-optimization advice gets backwards, and OpenAI's own latency guide is blunt about it. Trimming input is a cost win: "Cutting 50% of your prompt may only result in a 1-5% latency improvement." Trimming output is a time win: "cutting 50% of your output tokens may cut ~50% of your latency", or as the guide puts it, "an output token cut is a (milli)second earned!"

So if the complaint is spend, compress the context. If the complaint is that the agent feels slow, the levers are different ones: generate fewer output tokens, make fewer sequential requests, parallelize the steps that do not depend on each other, stream so the user waits less, and, first of all, do not default to a model for work a lookup table can do.

There is a structural reason for the asymmetry, and it is worth knowing so you stop being surprised by it. Input tokens are processed in parallel in a single forward pass; output tokens are generated one at a time, each waiting on the last. Halving the prompt shortens one prefill. Halving the response removes half the sequential steps. So "be concise" in a system prompt, a tighter output schema, and shorter field names in JSON are latency work, not cosmetic work, and they are usually cheaper to ship than a retrieval redesign.

Paying for speed directly is also now an option worth pricing. Anthropic's fast mode, in research preview, charges $8 per million input and $40 per million output on Claude Opus 5.5 against $4 and $20 at standard speed. That is a clean 2x for latency with no quality change, which makes it a useful upper bound on what your own engineering time is worth: if a week of pipeline work would save less than the fast-mode premium on your volume, buy the speed.

Measuring the Wins

Optimizations you cannot measure regress quietly. Three numbers matter: tokens split into input and output, because they price differently; wall-clock latency per request; and the fraction of requests that reach the expensive model.

Use the OpenTelemetry GenAI semantic conventions rather than inventing attribute names. gen_ai.operation.name and gen_ai.provider.name are Required, gen_ai.request.model is Conditionally Required, and gen_ai.usage.input_tokens, gen_ai.usage.output_tokens, gen_ai.usage.cache_read.input_tokens and gen_ai.usage.cache_write.input_tokens are Recommended. All of them are still marked Development stability, so pin the instrumentation version.

import time
from collections import defaultdict

stats = defaultdict(lambda: {"calls": 0, "in": 0, "out": 0, "cached": 0, "secs": 0.0})

def track(stage: str, model, prompt: str) -> str:
    t0 = time.perf_counter()
    resp = model.invoke(prompt)
    u = resp.usage_metadata or {}
    s = stats[stage]
    s["calls"] += 1
    s["in"] += u.get("input_tokens", 0)
    s["out"] += u.get("output_tokens", 0)
    s["cached"] += (u.get("input_token_details") or {}).get("cache_read", 0)
    s["secs"] += time.perf_counter() - t0
    return resp.content

The cache-read column is the one to watch alongside the expensive-model ratio. A cache hit rate that drops to zero after a deploy usually means someone moved a dynamic value into the prompt prefix, and the bill notices before anyone else does.

Which Framework Should You Use?

All three express the same playbook. The choice is about the shape of your problem.

LangChain suits a mostly linear agent where the wins come from middleware you configure rather than control flow you design. LangGraph earns the state object once you need retries, fallback routing and node caching you can reason about. The multi-agent route fits when the problem genuinely is a team of specialists, and if you are starting there today the live options are AG2 or Microsoft Agent Framework rather than AutoGen itself.

We would point most teams at LangChain first to validate the pattern, then graduate the same flow into LangGraph once branching and durable state matter. The planner, the retrieval guard and the compressor carry over nearly unchanged.

Where Fabriq Fits

Cost attribution needs to know who a call belonged to, and that is an identity problem rather than a billing one. Every call through Agentic Fabriq carries two identities, the agent and the acting user, and lands an audit record with both, which gives you the join key for attributing spend later. That is aact:audit-granularity/per-action-with-principal-chain in the Agent Access Control Taxonomy, where each action records the agent and the human it acted for rather than just that a session happened. We do not total up the money for you; what we provide is the record that says which agent acted for which person, with credentials attached server-side so the agent never handles the secret.

FAQ

Is a smaller model actually good enough to plan? For enumerating steps, triaging, and compressing retrieved text, usually yes, and the price spread means you can afford to check rather than assume. Hold an eval set fixed and move one stage at a time to the cheap tier.

Does prompt caching help latency or only cost? Both, but the cost effect is the one with published numbers. Treat the latency improvement as a bonus you measure rather than a figure you quote.

Why is my cache hit rate zero? Almost always a prefix that is not byte-identical. OpenAI's docs require "the entire rendered prefix" to match, tool definitions and developer messages included, so a timestamp, a randomized tool order, or a per-user greeting at the top will do it.

Should I use set_llm_cache? LangChain 1.x's model documentation covers provider-level caching and prompt-caching middleware rather than the old global cache. Prefer the provider cache: it is keyed on the prefix rather than the whole request, so it still hits when the last message differs.

What about batching? If the workload can tolerate asynchronous completion, both providers publish a 50% discount on input and output, and it is the largest single discount available. Most agent workloads described as real-time are not.

Conclusion

Three frameworks, one pattern, and a set of numbers that do not care which framework you picked. Route cheap reversible work to a small model, keep your prompt prefix stable enough to cache, skip the work nothing needs, and keep the output short because that is the token class that costs both money and time.

The part worth remembering is the asymmetry. Compression is a cost lever with a 1-5% latency effect, and output length is a latency lever with a proportional effect. Teams that conflate the two spend a quarter shrinking prompts and wondering why the agent still feels slow.

Sources

All URLs read 29 September 2026.

• Latency optimization, OpenAI — the 1-5% input versus ~50% output latency figures, "an output token cut is a (milli)second earned!", and the seven levers including not defaulting to an LLM.
• Prompt caching, OpenAI — the 1,024-token minimum, the 0.1x cached rate, the 30-minute window on GPT-5.6 and later, the entire-rendered-prefix rule, and usage.input_tokens_details.cached_tokens.
• Pricing, OpenAI — per-million input, cached input and output rates, and the Batch tables at half of standard.
• Pricing, Claude platform docs — model rates, the 1.25x/2x/0.1x cache multipliers, the 50% Batch discount, fast-mode pricing, the tool-use system prompt token counts, and the note that Claude 4.7 and later use a tokenizer producing about 30% more tokens.
• Prompt caching, Claude platform docs — minimum cacheable lengths by model, four cache breakpoints, the 20-block lookback, and the break-even rule for 5-minute and 1-hour entries.
• Built-in middleware, LangChain docs — SummarizationMiddleware, ContextEditingMiddleware, ModelCallLimitMiddleware, ToolCallLimitMiddleware, LLMToolSelectorMiddleware, ProviderToolSearchMiddleware and ModelFallbackMiddleware.
• Middleware, LangChain docs — how middleware attaches to create_agent.
• Models, LangChain docs — implicit provider caching, cache_control and prompt_cache_key, the prompt-caching middleware, and cache_read in input_token_details.
• Graph API, LangGraph docs — CachePolicy with key_func and ttl, compile(cache=InMemoryCache()), the {'cached': True} metadata, add_conditional_edges, and the 1,000-step recursion limit from 1.0.6.
• AutoGen repository README — the maintenance-mode notice and the pointer to Microsoft Agent Framework.
• AutoGen to Microsoft Agent Framework migration guide — single-turn AssistantAgent versus multi-turn Agent, and the orchestration differences.
• AG2 quick start — the ag2 install name and the current import path.
• GenAI span semantic conventions, OpenTelemetry — token-usage and model attribute names, requirement levels, and Development stability.
• PyPI release records — versions and release dates for langchain, langgraph, ag2, autogen-agentchat and agent-framework as at 29 September 2026.
• Agent Access Control Taxonomy — the audit granularity dimension, and the question it asks: what is recorded about agent actions, and who they are attributed to.