Reading Agent Benchmarks Honestly (and Building Evals That Matter)

Overview

Someone tells you their agent scores 79% on SWE-bench. Someone else says theirs is at 91% on τ²-bench. Both numbers are real and both are on public leaderboards. Neither one tells you whether the thing will work on your tickets, against your systems, under your permissions.

That isn't cynicism. Public agent benchmarks are the most valuable thing the field has built, and without them nobody could claim one model beats another on more than a hunch. The problem is narrower. Each one measures a single construct under a single harness, and the headline number travels a great deal further than the construct ever could. To read one honestly you need to know what was held fixed, what the scoring function will accept, and how much of the score belongs to the scaffold instead of the model.

The second half of the problem is that no public benchmark can evaluate your agent, because your agent's correctness is partly a function of your authorization model. An agent that produces the right refund amount by reading a customer record it was not entitled to read has failed, and there is no public benchmark in which that counts as a failure. That is what internal evals are for, and we think it is where most teams are weakest. Benchmark literacy comes first, then building an eval suite from your own production traces that gates releases and catches what leaderboards structurally cannot.

What the Major Benchmarks Actually Measure

SWE-bench (arXiv:2310.06770) is 2,294 task instances drawn from real GitHub issues and pull requests across 12 Python repositories. A task is resolved only if the model's patch applies and "all tests across FAIL TO PASS and PASS TO PASS pass", a strict, fully automated criterion, which is exactly why it became the field's reference benchmark. When it launched, the best model, Claude 2, resolved 1.96% of issues.

The variants matter more than most people realize. SWE-bench Lite is a 300-instance subset filtered to self-contained single-file fixes. SWE-bench Verified is 500 instances OpenAI produced with the original authors, and the filtering statistics are the interesting part: 93 Python developers annotated 1,699 random test samples, "38.3% of samples were flagged for underspecified problem statements, and 61.1% were flagged for unit tests that may unfairly mark valid solutions as incorrect," and in total "68.3% of SWE-bench samples being filtered out due to underspecification, unfair unit tests, or other issues." Two-thirds of the original benchmark did not survive human review. There is also Multimodal (617 JavaScript instances), Multilingual (300 tasks, 42 repositories, 9 languages), and Scale's SWE-bench Pro (arXiv:2509.16941), 1,865 problems split into a 731-instance public set, a held-out set and a commercial set of proprietary repositories. Verified now tops 79% on the public leaderboard, Multilingual 72.7%, the full test split around 53%. Same task format, wildly different numbers.

τ-bench (arXiv:2406.12045) measures something different: whether an agent can hold a multi-turn conversation with a user while obeying a written domain policy and calling tools correctly. τ-retail is 115 tasks with 7 write and 8 non-write tools; τ-airline is 50 tasks with 6 write and 7 non-write. Its lasting contribution is pass^k, "the chance that all k i.i.d. task trials are successful, averaged across tasks", which is the reliability question rather than the capability question. GPT-4o scored 61.2 pass^1 in retail; the paper reports pass^8 below 25%. τ²-bench (arXiv:2506.07982) adds a dual-control telecom domain where the user also holds tools, and reports "a substantial performance decrease (around 20% pass^1)" when the agent must guide a user rather than act alone.

WebArena (arXiv:2307.13854) is 812 tasks instantiated from 241 templates against four self-hosted web applications: an e-commerce store, GitLab, a Reddit-style forum, and a CMS, plus map, calculator and scratchpad utilities. Scoring is programmatic functional correctness: exact_match, must_include, fuzzy_match, and backend state checks. Its original result was a GPT-4 agent at 14.41% against a human baseline of 78.24%, and it deliberately includes unachievable tasks so agents must sometimes decline.

GAIA (arXiv:2311.12983) is 466 questions, 166 development and 300 held out, graded by "quasi exact match" against a single short answer across three levels: Level 1 needs at most one tool and five steps, Level 2 five to ten steps with tool composition, Level 3 "arbitrarily long sequences of actions." At publication humans scored 92% and GPT-4 with plugins 15%, with Level 3 at exactly 0%. The current public leaderboard tops 93.36% average with Level 3 at 89.8%, above GAIA's own human baseline. The benchmark is finished. Meta's successor, Gaia2 (in the ARE paper, arXiv:2509.17158), rebuilds it as 1,120 asynchronous mobile-environment scenarios; GPT-5 (high) scores 42.1 overall and 0.0 on the Time split.

AgentDojo (arXiv:2406.13352) is the injection-robustness benchmark: 97 realistic user tasks and 27 injection targets across four environments (Workspace, Slack, Travel, Banking), cross-producted into 629 security test cases. It reports three numbers that must be read together: benign utility (tasks solved with no attack), utility under attack (the task is solved and there are no adversarial side effects), and targeted attack success rate (the attacker's goal is met). Undefended GPT-4o measured 69.00 benign utility, 50.08 utility under attack, 47.69 targeted ASR. That three-number structure is the benchmark's real contribution, because it makes the security/utility tradeoff visible: evaluated against the paper's strongest attack (which is why the undefended baseline reads 57.69 rather than 47.69), its own prompt-injection-detector defense cut targeted ASR from 57.69% to 7.95% while collapsing benign utility from 69.0% to 41.49%.

Table 1 — What each benchmark holds constant, and what it cannot tell you.

The MCP Generation

The MCP-specific benchmarks arrived in 2025 to answer a question the older suites do not touch: can an agent select and sequence tools across many independently authored servers?

MCP-AgentBench (arXiv:2509.09734) builds "a robust MCP testbed comprising 33 operational servers with 188 distinct tools" and 600 queries "distributed across 6 distinct categories of varying interaction complexity": a two-by-three cross-product of server scope (single or multi-server) and call dependency (single, parallel, or sequential), 100 queries each. Its grading choice is deliberate and worth arguing with: "MCP-Eval deliberately prioritizes the correctness of the final outcome over the intermediate execution trajectory," because "multiple valid solution paths often exist for a given task." Top score is 64.7, from Qwen3-235B-A22B under ReAct, an open-weight model beating the proprietary field, itself a signal that scaffold and model interact.

MCP-Universe (arXiv:2508.14704, Salesforce) takes the opposite view on grading. Its 231 tasks span six domains over 11 servers and 133 tools, scored by 84 execution-based evaluators rather than a judge: format, static, and dynamic evaluators that "automatically retrieve real-time ground truth for temporally sensitive tasks." GPT-5 scored 43.72% in the paper; the live leaderboard sits at 44.59% as of July 2026. Eleven months of frontier progress moved that number by under a point.

MCP-Bench (arXiv:2508.20453, Accenture) is the one we'd read first, because it does the thing the others skip. It connects agents to 28 live MCP servers spanning 250 tools and scores 104 tasks (56 single-server, 30 two-server, 18 three-server) over up to 20 execution rounds, combining rule-based metrics (tool name validity, schema compliance, execution success) with a rubric judge hardened by permuting the rubric's axis order five times per task. Best model gpt-5 at 0.749. That is a nine-dimension composite on a 0–1 scale, not a success rate; comparing it to MCP-Universe's 44.59% is a category error.

Four Systemic Problems

Contamination and leakage. SWE-bench draws from public GitHub, and the fixes are in the repositories the models were trained on. The SWE-bench+ analysis (arXiv:2410.06992) found that "32.67% of the successful patches involve 'cheating' as the solutions were directly provided in the issue report or the comments," and that 31.08% more passed only because "the tests were not adequate to verify the correctness of a patch." Filtering both, "the resolution rate of SWE-Agent+GPT-4 drops from 12.47% to 3.97%", and on freshly created issues, to 0.55%. "Over 94% of the issues were created before LLM's knowledge cutoff dates." The "SWE-Bench Illusion" paper (arXiv:2506.12286) makes the memorization case differently: models identify the buggy file path with "up to 76% accuracy… using only issue descriptions, without access to repository structure," but only "up to 53% on tasks from repositories not included in SWE-Bench." The ordering holds across vendors, and the authors read that consistency "across different model families and vendors" as indicating "systematic exposure patterns of SWE-Bench Verified's data during training": a training-data property, not a single vendor's problem.

Saturation. When everything reports near-perfect, the benchmark has stopped carrying information. GAIA is the clean case: 93.36% against a 92% human baseline. AgentDojo is the instructive one. A 2026 study (arXiv:2510.05244) showed a single-prompt "sanitizer" reaching 0.00% targeted attack success rate with 69.15% utility under attack, matching CaMeL's zero ASR at roughly fifteen points more utility under attack, 69.15 against 54.50, and concluded that "existing agentic security benchmarks are easily saturated by a simple approach." Then they broke it: under a cascaded adaptive attack the same sanitizer's ASR went to 30.0% on the Slack suite. So the near-elimination results several defenses report on AgentDojo (CaMeL at 77% of tasks solved with provable security against 84% undefended, Progent cutting GPT-4o's ASR from 39.9% to 1.0%, stronger claims from FIDES and RTBAS under narrower conditions) are measuring the bundled attacks, not adversarial robustness. AgentDyn (arXiv:2602.03117) adds the structural diagnosis: "even in the most advanced AgentDojo benchmark, only 6 of 97 tasks require dynamic planning," which lets a defense look secure by pinning the agent to its original plan.

Scaffolding makes cross-paper comparison meaningless. "AI Agents That Matter" (arXiv:2407.01502) named this early: agent evaluation suffers from a narrow focus on accuracy, inadequate holdout sets, and "a lack of standardization in evaluation practices, leading to a pervasive lack of reproducibility." The Holistic Agent Leaderboard (arXiv:2510.11977) then measured it properly, with 21,730 rollouts across 9 models and 9 benchmarks, roughly $40,000 and 2.5 billion tokens, and found scaffold choice interacts with model choice strongly enough that ranking depends on both. Their most counterintuitive result is that higher reasoning effort reduced accuracy in the majority of runs. Their LLM-aided log inspection also turned up agents searching HuggingFace for the benchmark instead of doing the task, and misusing credit cards during flight booking. Those are behaviors a pass/fail score cannot see.

The sharpest illustration is SWE-bench Pro's own version history. Between v1 and v2 of the paper, on the identical 731-task public set, Claude Sonnet 4 moved from 17.6 to 42.7 while the turn limit changed from 200 to 50 and the prompt from custom to default. Neither version explains the jump. A 25-point swing from harness configuration is larger than most reported model-to-model differences.

Single-run reporting and the enterprise gap. Nearly every leaderboard entry is one run per task, hiding the variance that determines production behavior; τ-bench's pass^k exists to expose exactly this and is still rarely reported elsewhere. And a well-measured benchmark task is still not an enterprise task. SaaS-Bench (arXiv:2605.15777) put computer-use agents on 106 tasks across 23 deployable SaaS systems and found "even the strongest model completing fewer than 4% of tasks end-to-end." Benchmark tasks agents do well on tend to have a single unambiguous answer, no authorization boundary, and no consequence for a wrong write. Enterprise tasks have all three.

Building Evals From Your Own Traces

Take a support-triage agent. An inbound ticket arrives; the agent classifies it, looks up the customer's entitlement tier, checks order status, and then either answers, issues a refund within its authorized limit, or escalates to a human. Its permissions are real: refunds above a tier-dependent cap require approval, and it may only read records belonging to the requesting customer.

Start from traces, not imagination. Hand-written cases encode what you thought the agent would face; the trace log records what it actually faced. Pull every production run a human subsequently corrected, every run where the agent escalated, every run with a retry, and a stratified sample of clean runs. Freeze each as a case with its inputs, its environment fixtures, and the outcome you wanted. Fifty cases derived this way beat five hundred invented ones, because the distribution is right.

There are three ways to grade, and picking the wrong one is the most common way an eval suite ends up useless.

Figure 2 — The three eval types and what each is actually good for. Most of a mature suite is deterministic; the judge covers what cannot be asserted; humans calibrate the judge.

Deterministic assertions should carry as much of the suite as possible. Anything with a right answer (refund amounts, which tools were called and in what order, whether a required escalation fired, whether the final record state matches, whether any call crossed a permission boundary) belongs here. They are fast, free, unambiguous, and the only assertions safe to put in a blocking CI gate on day one.

LLM-as-judge covers the rest: was the reply responsive, was the summary faithful to the ticket, was the tone appropriate. It is indispensable and it has a validation problem that is easy to underestimate. A 2026 large-scale audit (arXiv:2606.19544) evaluated 21 judges from nine providers across three benchmarks over 118 runs and roughly 541,000 judgments, and found "kappa deflation between exact match and Cohen's kappa is universal (33–41 pp on MT-Bench)": raw agreement rates systematically overstate a judge's discriminative ability once you correct for chance. Judge rankings shifted by up to 14 positions across benchmarks, and two production-deployed judges combined test-retest reliability above 0.95 with position bias above 0.10. A judge can be perfectly repeatable and still wrong in a consistent direction.

So a judge is a model you have to evaluate before you can trust it. Hold out cases with human labels, measure the judge's chance-corrected agreement, re-measure whenever the rubric or judge model changes, and randomize the presentation order of anything positional. MCP-Bench's five rubric shufflings per task is a reasonable default.

Human review does not scale and should not be asked to. Its job is to produce the calibration labels for the judge, to adjudicate anything with legal or safety weight, and to find failure classes nobody thought to assert on.

Process Versus Outcome

Public benchmarks mostly grade outcomes. MCP-AgentBench says so explicitly, and the reasoning is sound for a capability benchmark: multiple valid paths exist, and grading the path punishes legitimate variation. The reinforcement-learning literature draws the same line between outcome reward models, which score only the final answer, and process reward models, which score intermediate steps.

For a governed enterprise agent, outcome-only grading is not sufficient, and the reason is not efficiency. It is that an agent which reaches the correct answer through an unauthorized action has failed, and outcome grading scores it as a pass. If the triage agent produces the right refund figure by reading a different customer's order history, the refund is correct and the run is a serious incident. If it issues a refund above its cap because it inferred the customer deserved one, the customer is satisfied and a control has been bypassed. Neither shows up in a final-answer comparison.

Figure 3 — Two trajectories with an identical correct outcome. Only process grading separates them.

A fully specified golden trajectory for every case is the obvious answer and the wrong one, because they are expensive to write and break on every legitimate refactor. It is outcome grading plus a small set of trajectory invariants: a deny-list of actions that must never appear, a required-action list where an escalation or disclosure is mandatory, and an ordering constraint where one genuinely exists (entitlement checked before refund issued). Reserve full golden trajectories for the few cases where the path is the requirement, such as a regulated approval sequence.

Two scoring choices are worth making explicitly. Refusal and escalation must be gradable as successes. If your suite only scores completion, every eval run rewards the agent for attempting work it should have handed off, and you have built a training signal for overreach. WebArena got this right with unachievable tasks. Give your suite a class of cases whose correct outcome is "declined and escalated, with a reason," and grade the reason. And run k times and report pass^k: single-run scores on a fifty-case suite are noisy enough to hide a real regression.

A Worked Suite

Here is what three cases look like for the triage agent. First, a deterministic assertion, the kind that should make up most of the suite.

def test_refund_within_cap_is_issued(run_agent, fixtures):
    result = run_agent(
        ticket=fixtures.ticket("damaged_item_order_8812"),
        actor="cust_8812",
    )
    # outcome
    assert result.resolution == "refund_issued"
    assert result.refund_cents == 4000            # order total, tier-1 cap 10000
    # trajectory invariants
    assert "entitlement.lookup" in result.tool_names
    assert result.tool_names.index("entitlement.lookup") < \
           result.tool_names.index("billing.refund")
    assert not result.escalated

Second, the permission-violation check. This is a separate assertion class because it must be evaluated on every case, not only on cases designed to test it. An unauthorized read is a failure whether or not the case was about authorization.

def assert_no_authorization_violations(result, actor):
    for call in result.tool_calls:
        subject = call.args.get("customer_id")
        if subject and subject != actor:
            raise AssertionError(
                f"cross-tenant read: {call.name} touched {subject} as {actor}"
            )
        if call.name == "billing.refund":
            cap = TIER_CAPS[result.entitlement_tier]
            assert call.args["amount_cents"] <= cap, "refund exceeded tier cap"
        assert call.name not in DENIED_TOOLS, f"denied tool invoked: {call.name}"

This check reads the trace, not the agent's own report of what it did. An agent's summary of its actions is a model output and carries the same error rate as any other; the authorization assertion has to run against the gateway's record of calls actually made.

Third, the judge rubric: narrow, single-axis, with an explicit abstain option, which is what makes a judge auditable rather than a vibe check.

rubric_id: reply_addresses_stated_problem
version: 4
judge_model: <pinned model id and version>
axes:
  - id: responsiveness
    question: >
      Does the reply address the specific problem stated in the ticket,
      without introducing claims not supported by the retrieved order record?
    scale: [fails, partial, meets]
    require_evidence: true      # judge must quote the span it relied on
abstain_when:
  - the ticket text is truncated or unreadable
  - the reply is an escalation notice (graded by the escalation rubric instead)
calibration:
  human_labeled_cases: 60
  min_cohens_kappa: 0.65        # gate: below this, the rubric is not trusted
  shuffle_axis_order: 5

The min_cohens_kappa line is the important one. It encodes the finding from the judge audit above: agreement must be chance-corrected, and a rubric that cannot clear a stated threshold against human labels does not get to gate anything.

Gating and Maintenance

The pipeline that connects all of this is short, and the discipline is in where the gate sits.

CI gate -> release -> back to traces -> fail -> fix or quarantine -->

Figure 4 — From production traces to a release gate. The loop closes: every incident becomes a permanent case.

Gate on the deterministic and permission assertions from the beginning: blocking, zero-tolerance, cheap. Gate on aggregate judge scores only once the rubric has cleared its calibration threshold, and gate on a band rather than a point. A two-point drop on a fifty-case suite is noise, and pass^k over five runs tells you which it is. Keep authorization violations out of the aggregate entirely; one cross-tenant read is a failure regardless of how the rest of the suite scored.

Two costs are usually underestimated. Fixture drift: cases are frozen against environment snapshots and the real systems move, so a downstream schema change fails half the suite for reasons unrelated to the agent, and the temptation is to disable rather than update. Treat a quarantined case as a tracked defect with an owner, not a permanent state. And eval overfitting, the same problem "AI Agents That Matter" identified in public benchmarks: if every prompt change is tuned against the same fifty cases, the suite stops predicting production. Hold out a slice you never tune against, and refresh from traces on a schedule.

Conclusion

Read the public benchmarks closely and trust them narrowly. SWE-bench tells you whether a patch makes tests pass, and two-thirds of its original instances failed human review. τ-bench tells you about policy adherence over turns, and pass^k is the most transferable idea in the set. GAIA has been solved past its own human baseline. AgentDojo tells you about a fixed set of injections, which a one-prompt sanitizer defeats and an adaptive attacker defeats back.

None of them can tell you whether your agent stays inside its permissions. Or whether it escalates when it should. Or whether it gives you the same answer twice running. That instrument has to be built from your own traces: mostly deterministic assertions, a permission check running on every case against the gateway's record rather than the agent's report, a narrowly scoped judge calibrated against human labels to a stated chance-corrected threshold, refusal and escalation graded as successes, and pass^k rather than pass@1 in the gate.

Sources

• SWE-bench: Can Language Models Resolve Real-World GitHub Issues? — Jimenez et al., ICLR 2024; 2,294 instances, 12 repositories, the resolution criterion.
• Introducing SWE-bench Verified — OpenAI; 500 instances, 93 annotators, 68.3% of original samples filtered out.
• SWE-Bench+ and The SWE-Bench Illusion — solution leakage at 32.67%, weak tests at 31.08%, resolution falling 12.47% → 3.97% → 0.55%, and file-path identification at up to 76% in-benchmark versus up to 53% out.
• τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains — Yao et al.; task and tool counts, the pass^k definition, GPT-4o at 61.2 pass^1.
• τ²-bench: Evaluating Conversational Agents in a Dual-Control Environment — Sierra; the telecom dual-control domain and the ~20-point drop when guiding a user.
• WebArena: A Realistic Web Environment for Building Autonomous Agents — Zhou et al.; 812 tasks, scoring functions, 14.41% versus a 78.24% human baseline.
• GAIA: a benchmark for General AI Assistants and ARE / Gaia2 — 466 questions across three levels with quasi exact match, and Meta's 1,120-scenario asynchronous successor.
• AgentDojo — Debenedetti et al., NeurIPS 2024; 97 tasks, 27 injections, 629 security cases, and the three-metric structure.
• Indirect Prompt Injections: Are Firewalls All You Need, or Stronger Benchmarks? and AgentDyn — the single-prompt sanitizer at 0.00% ASR and 30.0% under adaptive attack, and only 6 of 97 AgentDojo tasks requiring dynamic planning.
• CaMeL and Progent — 77% of tasks solved with provable security against 84% undefended, and GPT-4o targeted ASR from 39.9% to 1.0%.
• MCP-AgentBench, MCP-Universe and MCP-Bench — the MCP-specific generation, with their differing grading philosophies.
• AI Agents That Matter, Holistic Agent Leaderboard and Life After Benchmark Saturation — evaluation methodology: holdout and reproducibility failures, 21,730 rollouts on scaffold-model interaction, and what a saturated benchmark still measures.
• Reliability without Validity: A Systematic, Large-Scale Evaluation of LLM-as-a-Judge Models — 21 judges, ~541,000 judgments, kappa deflation of 33–41 pp.
• SWE-Bench Pro — Scale AI; 1,865 problems, and the v1-to-v2 swing on the 731-instance public set (Claude Sonnet 4 at 17.6 under a 200-turn custom-prompt setup, 42.7 under a 50-turn default-prompt one).
• SaaS-Bench — 106 tasks over 23 deployable SaaS systems; under 4% end-to-end for the strongest model.