Why Your Agent Works in the Demo and Fails in Production

TL;DR

Your agent's success rate is set by how many steps it takes, not by how good the model is. Eighteen steps at 95% each lands at 40%, and no amount of prompting changes the exponent.

Nothing has to be broken for that to happen. Every step survives review. Every step is defensible on its own. The product is still 0.397.

Which is why upgrading the model usually disappoints. You moved one factor out of eighteen.

Retries help less than people hope. They're good against timeouts and close to useless against a stale answer, because a stale answer looks exactly like a fresh one.

You can work your own number out in a minute. Count the steps, estimate the per-step rate, multiply. Most teams never do, and hear it from their users instead.

Overview

Take an accounts-payable agent, the kind that picks up invoices failing three-way match. It fetches the invoice, the purchase order and the goods receipt. It normalizes line items across three schemas that disagree about almost everything. Then variance, tolerance lookup, duplicate check, a decision about whether to auto-approve or adjust or escalate, a write to the ERP, a status update, an audit note, and emails to both the clerk and the vendor. Eighteen steps, give or take.

In the demo it works. It works again on a second invoice, which is usually where people stop testing. Then it meets four thousand exceptions a month and starts producing things nobody designed: invoices marked resolved that aren't, adjustments posted twice, escalations that quietly didn't escalate.

The first instinct is nearly always the model. So you upgrade it, and the numbers twitch, and the failure modes don't move at all. That last part is the tell.

The model is rarely the interesting part. What sets your failure rate is how many steps you asked for and how reliable each one is, and a demo is about the worst instrument you could pick for measuring either, being one run against clean fixtures with a cooperative user.

The Arithmetic of Compounding

95% per step sounds fine. It's the kind of number that sails through a design review without anyone squinting at it. Now make all eighteen steps land:

0.95^18 ≈ 0.397

Roughly 40% of invoices come out right. At twenty steps it's 35.8%. The 95% was never wrong. It was being read as though it described the job, when it describes one step of the job.

Table 1 — End-to-end task success as a function of per-step reliability and task length, assuming independent steps.

Read the table backwards and it turns into a requirement. Want 90% end-to-end across twenty steps? Every step has to hit 0.9^(1/20) ≈ 99.5%. At fifty steps you need 99.8%. You don't get numbers like that from better prompting. You get them by deleting steps, by making the ones left deterministic, or by catching failures before they travel.

wrong state --> silent wrong outcome Every step is a branch point. The correct path is the product of all of them. -->

Figure 1 — Compounding is the product of the per-step success probabilities. Each 5% branch is a different failure, and most of them do not announce themselves.

Independence is an approximation in both directions. Steps sharing a bad assumption fail together; steps operating on state an earlier step validated correlate positively. It gets the shape right anyway: end-to-end reliability decays roughly exponentially in horizon length. That shape has empirical support. Toby Ord's analysis of METR's task-duration data (arXiv:2505.05115) shows agent performance across METR's research-engineering suite (arXiv:2503.14499) is well explained by "a constant rate of failing during each minute a human would take to do the task": an exponential survival curve with a per-agent half-life. Horizon is the independent variable, and reliability falls off it fast.

What pass^k Measures That pass@1 Hides

Compounding is the mechanism. Single-run benchmark scores are what keep it hidden. You run each task once, write down pass or fail, average the column, and what you've measured is how the agent does on a good day. Not whether it'll do it again on Tuesday.

τ-bench (arXiv:2406.12045, Yao et al.) introduced the metric that fixes this. Its two domains (τ-retail with 115 tasks and 15 tools, τ-airline with 50 tasks and 13 tools) put an agent in a multi-turn conversation with a simulated user and a domain policy it must follow. Alongside average success it defines pass^k: the probability of succeeding on all k independent trials of the same task, estimated as E_task[ C(c,k) / C(n,k) ] for a task attempted n times with c successes.

pass@k and pass^k point in opposite directions. pass@k asks whether at least one of k attempts succeeded, which is a useful question when a human reviews candidates and picks the good one. pass^k asks whether every attempt succeeded. Production users do not get k attempts and do not pick; they get whatever the agent did the first time, and if the invoice is one of the 40% that goes wrong, the wrongness is already in the ERP.

The gap on τ-bench is large. GPT-4o scores 61.2% pass^1 in retail; the paper reports pass^8 below 25%.

Running k times lowers your score, which is why nobody wants to do it. The useful part is that it sorts your tasks. Some come back reliable and can ship today. Some are unreliable, and those are the ones the rest of this piece is about. The rest are out of reach, and that's a scoping conversation, not an engineering one. Have it early.

Figure 2 — pass@1 and pass^k on the same three tasks. The middle band is where production incidents come from, and it is the band a single-run score describes least well.

There is a second reason single-run scores overstate readiness, and it concerns the user. τ-bench's user is a language model following a scripted persona. A 2026 study of that assumption (arXiv:2603.11245) ran the full τ-bench protocol with 451 real human participants across all 165 tasks and compared it against 31 LLM user simulators. Real humans drove a 63.6% agent success rate; the strongest simulators drove up to 77.8%. The authors describe LLM simulators as excessively cooperative and stylistically uniform, an "easy mode" that inflates agent success above the human baseline. Sierra's follow-on τ²-bench (arXiv:2506.07982), which lets the user act on the shared environment rather than just talk, reports "a substantial performance decrease (around 20% pass^1) when agents must shift from autonomous operation to guiding a user." Distribution shift in how people phrase things is worth ten to fifteen points before any of the compounding math applies.

Reliability as a Property You Can Measure

The 2026 literature has been working on this, and the useful idea to take from it is that capability and reliability are not the same property and should stop sharing a number.

Rabanser et al., Towards a Science of AI Agent Reliability (arXiv:2602.16666), put it plainly: "compressing agent behavior into a single success metric obscures critical operational flaws." What gets obscured is whether agents "behave consistently across runs, withstand perturbations, fail predictably, or have bounded error severity." They borrow twelve metrics from safety-critical engineering across four dimensions, which is more apparatus than most teams will ever use. Two of their findings are worth your time. Across all 15 models they tested on GAIA and τ-bench, outcome consistency is low, and "recent capability gains have only yielded small improvements in reliability." And prompt robustness, which just means whether the agent survives you rephrasing the instruction, still separates models badly, even where fault and environment robustness have flattened out. That second one should worry anyone who tuned their prompts by hand and then stopped.

Khanal, Tao, and Zhou take on the horizon question in Beyond pass@1: A Reliability Science Framework for Long-Horizon LLM Agents (arXiv:2603.29231), running 10 models across 23,392 episodes on a 396-task benchmark stratified into four duration buckets and three domains. They define a Reliability Decay Curve, a Variance Amplification Factor for run-to-run spread, a Graceful Degradation Score, and a Meltdown Onset Point marking where behavior becomes catastrophic rather than merely worse. Decay turns out to be domain-stratified rather than universal: software-engineering GDS falls from 0.90 to 0.44 across duration buckets while document processing is nearly flat, 0.74 to 0.71. Frontier meltdown rates run as high as 19%. And past a certain horizon the capability ranking and the reliability ranking stop agreeing with each other, which should give pause to anyone choosing a model off a leaderboard. One of their results cuts against the obvious reading of spread: VAF bifurcates by capability tier, and they argue high VAF is "a capability signature, not an instability signal," because frontier models attempt ambitious multi-step strategies that sometimes spiral. Variance is a thing to explain, not a number to minimize.

Which is why variance belongs in the report next to the mean, rather than averaged out of it. An agent with 70% mean success and low variance is something you can build a process around. An agent with 70% mean success and high variance is a different system, and the two are indistinguishable on a leaderboard. Closing that gap is what Rabanser et al.'s consistency dimension is for.

A Taxonomy of Production Failure

Compounding gives you the rate. It doesn't tell you what actually went wrong, and the fixes are different depending on which one it was. Seven keep showing up.

Figure 3 — Seven production failure modes, grouped by where you have to go to fix them. Step-level failures get designed out. Control-loop failures belong to the runtime. Environment failures mostly come down to writing a contract and then checking whether anything honours it.

Compounding error is Table 1 made concrete. Nothing is individually broken; the product is simply small. The tell is that no single step looks like the culprit in a post-mortem, because each is defensible in isolation.

Recovery failure costs the most. The agent takes a wrong step and just carries on as if it hadn't. The optimistic reading is that a self-critique pass catches it; the evidence says otherwise. Huang et al. (arXiv:2310.01798, ICLR 2024) found that in the intrinsic setting, with no external feedback, models struggle to self-correct reasoning and sometimes get worse after trying. An agent checking its own work runs the same distribution over the same context that produced the error. Detection has to come from outside the loop: a schema validator, a re-read of written state, a reconciliation query, a second system's view of the same fact.

Context degradation over long runs is a distinct failure with a distinct fix, treated separately in a companion post on context engineering. Here it means per-step reliability is not constant: step 40 is less reliable than step 4 because it is taken with a window full of accumulated tool output. Modeling p as fixed is optimistic for long runs.

Tool and environment drift is the demo-to-production delta. The demo ran against three clean fixtures. Production returns a fourth line-item type, a vendor record with a null tax ID, a 504 under month-end load, and a goods receipt that posts eleven minutes after the invoice. AgentCheck (arXiv:2607.11098) makes this measurable by injecting twelve fault classes into tool responses: execution faults (timeout, API error, permission denial, schema drift), data-quality faults (stale data, contradictions, wrong answers, silent empty results), and security faults. Across five agent configurations on 120 scenarios, scores ranged from 105/120 to 77/120, and data-quality was the weak category for every one; even the strongest managed 29/40. The asymmetry is the part worth remembering: adding a retry took the weakest agent from 30% to 100% on timeouts, while stale-data faults stayed at 30–40% regardless of mitigation, because a stale answer is a plausible answer and there is nothing in it to retry on.

105 / 120

best agent configuration under injected faults

77 / 120

worst configuration on the same scenarios

30-40%

stale-data handling, unmoved by any mitigation tried

Distribution shift in user phrasing is the sim2real gap above. Scripted evaluation users are cooperative, unambiguous, and do not change their mind at turn six.

Non-idempotent retries turn one failure into two. The agent posts the adjustment, the response times out, the agent retries, and the vendor is credited twice. This is ordinary distributed-systems hygiene arriving somewhere that was not built for it, made worse because the retry decision is taken by a model rather than a client library with a state machine. MCP's tool annotations do not help: the 2026-07-28 specification requires that clients "MUST consider tool annotations to be untrusted unless they come from trusted servers," so a server's claim that a tool is idempotent is a hint, not something to build a retry policy on.

Silent partial success is what makes the other six hard to find. When Errors Become Narratives (arXiv:2606.14589) is an eight-week longitudinal study of one production agent runtime: roughly 40 scheduled jobs across 8 LLM providers, defended by 4,286 unit tests and 827 governance checks. Across 22 fully post-mortemed incidents the author finds at least 28 manifestations of one meta-pattern: a failure whose error signal never reaches a human in actionable form. Incident latency ranged from 13 hours to 60 days and tracked the failure mechanism rather than code complexity. The longest-lived failures lived in the seams between components, where no test runs. Roughly 70% of incidents were caught by a human reading the agent's actual output, not by the tests or the audits; a retrospective audit of 15 incidents found 0% ex-ante prevention but 87% regression blocking, which is the sharpest line in the paper: audits are regression engines, not prediction engines. It is a single-system study by a single author and reads as a field report rather than a general result. We still think the mechanism is one every team running agents will recognize.

What Actually Raises Reliability

So, the three levers, and what each actually looks like when you build it.

Shorten the horizon. This is the one we'd reach for first. Teams hate it. It means building less agent than you told everyone you were building. Eighteen model-decided steps become four, with the deterministic parts moved into code. Fetching the invoice, the PO, and the receipt is a join, not a reasoning task. Computing variance is arithmetic. Checking it against a tolerance table is a lookup. Anthropic's guidance is blunt: find "the simplest solution possible, and only increas[e] complexity when needed," adding it "only when it demonstrably improves outcomes." An agent whose model-driven surface is classify the exception and choose between adjust, hold, and escalate has two steps to compound, not eighteen.

Checkpoint and verify at boundaries. Put verification where an error becomes expensive to unwind, generally just before any write and just after any state transition, and make it independent of the step it checks: a schema assertion, a re-read of the record the agent claims it wrote, a totals reconciliation. Resumable checkpoints also mean a failure at step fourteen resumes at fourteen rather than restarting the horizon, which is the difference between one compounding chain and two.

Verification is not free. The verifier-tax study (arXiv:2603.19328) found runtime safety mediation could intercept up to 94% of non-compliant actions while safe task-completion rates stayed below 5% in most settings, with post-interception recovery ranging from 21% down to near zero. Blocking a bad action is much easier than producing a good one afterwards, so budget for the recovery path: a guard that halts the agent and leaves the invoice half-adjusted has converted a wrong outcome into an inconsistent one.

Make side-effecting tools idempotent. Every write tool takes a caller-supplied idempotency key derived from values that do not change on retry (run id, step index, entity id) and returns the cached first result for a duplicate key within a bounded window. This is the Stripe pattern, and it predates agents by a decade. Where a downstream system will not cooperate, the gateway in front of it can hold the dedupe table.

Detect failure explicitly rather than inferring it. Give tools a contract that distinguishes the call failed, the call succeeded and found nothing, and the call succeeded and here is the result. An empty list and a failed query look identical to a model, and one means "no duplicate invoice exists" while the other means "the duplicate check did not run." MCP's error model supports the distinction: protocol errors for malformed requests, tool execution errors returned as results with isError: true, which clients "SHOULD provide to language models to enable self-correction."

Bound retries and vary the strategy. Three attempts, not thirty, and the second should not be a byte-identical replay of the first. Same call for a timeout; a narrower query for an empty result; a human for a permission denial. Unbounded uniform retry is how a transient fault becomes a budget incident.

Narrow the task rather than broadening the agent. The instinct after a failure is to hand the agent more tools and more latitude so it can handle the case that broke it. That raises branching factor and horizon at once and makes the next failure less diagnosable. The better move is usually the reverse: carve the failing case into its own path, or route it to a human. An agent that resolves 70% of exceptions reliably and escalates the rest beats one that attempts 100% and is correct 60% of the time, because the first has a knowable error rate. We would ship the 70%.

Instrumenting for It

Your tool-call latency dashboard will never tell you that your pass^8 is 20%. Here's what we'd wire up instead, and we'd do it early, because the day you need this data is the day it's too late to start collecting it. Run the same task k times. Five to eight is informative enough. Report pass^k alongside the mean, on a fixed suite derived from real production cases. Track variance explicitly, because two releases with identical mean success and different spread are different systems. Record per-step outcomes, not just terminal ones, so a falling end-to-end number can be attributed to a step rather than to the model; work on consistency as a testable statistical property (arXiv:2605.10516) argues trajectory-level consistency carries more diagnostic signal than terminal pass rates for exactly this reason. And log the negative space: attempts abandoned, tools that returned empty, verifications that failed and were retried into success. And keep one human periodically reading what the agent actually produced. Not a scalable control, but a good calibration for whether the scalable ones are measuring anything.

Conclusion

None of this is mysterious, and very little of it is about model quality. A demo is one draw against clean fixtures with a cooperative user. Production is thousands of draws against a moving environment and people who phrase things however they feel like phrasing them. What comes out the far end is the product of every per-step probability on the way through. At 95% over twenty steps that product is 36%, and you can't prompt your way out of an exponent.

What changes it, in our view, is treating reliability as something you measure and engineer: fewer model-decided steps, independent verification where errors get expensive, idempotent writes, explicit failure signals, bounded and varied retries, and a scope narrow enough that the agent's job is one it can do every time. Report pass^k rather than pass@1, treat variance as a metric rather than noise, and accept the partition it hands you.

Sources

• τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains — Yao, Shinn, Razavi, Narasimhan; introduces pass^k, τ-retail (115 tasks) and τ-airline (50 tasks).
• Towards a Science of AI Agent Reliability — Rabanser, Kapoor, Kirgis, Liu, Utpala, Narayanan; twelve reliability metrics across four dimensions, 15 models on GAIA and τ-bench.
• Beyond pass@1: A Reliability Science Framework for Long-Horizon LLM Agents — Khanal, Tao, Zhou; RDC, VAF, GDS and MOP over 23,392 episodes.
• Mind the Sim2Real Gap in User Simulation for Agentic Tasks — 451 human participants over 165 τ-bench tasks against 31 LLM user simulators.
• τ²-bench: benchmarking agents in collaborative real-world scenarios — Sierra; dual-control environments and the solo-to-interactive drop.
• Is there a half-life for the success rates of AI agents? — Toby Ord; constant per-minute failure-rate model.
• Measuring AI Ability to Complete Long Software Tasks — Kwa et al., METR; the 50%-task-completion time horizon.
• Large Language Models Cannot Self-Correct Reasoning Yet — Huang et al., ICLR 2024; limits of intrinsic self-correction.
• AgentCheck: A Reproduce–Intervene–Mitigate Workbench for LLM Agents over MCP — twelve injected fault classes, 120 scenarios, five agent configurations.
• When Errors Become Narratives: A Longitudinal Taxonomy of Silent Failures in a Production LLM Agent Runtime — eight-week single-system field study, 22 incidents, five failure classes.
• The Verifier Tax: Horizon-Dependent Safety–Success Tradeoffs in Tool-Using LLM Agents — interception rates versus safe completion rates.
• Consistency as a Testable Property: Statistical Methods to Evaluate AI Agent Reliability — trajectory-level consistency as a sharper diagnostic than pass@1.
• MCP specification 2026-07-28 — Tools — tool annotations as untrusted hints, and the protocol-error versus tool-execution-error distinction.
• Building Effective Agents — Anthropic; simplicity first, complexity only where it demonstrably helps.