Why Do BIRD and Spider Rankings Disagree?

Overview

Someone sends you two tabs. On one, a system is fourth. On the other, the same system is second and the thing that beat it is now seventh. Both tabs are text-to-SQL leaderboards, both are current, and nobody is lying.

The reason is that a leaderboard score has four inputs and only one of them is the system. The other three are the question set, the databases those questions run against, and the few dozen lines of Python that decide whether a returned table counts as the right answer. Spider 1.0, BIRD and Spider 2.0 differ on all three. Within a single benchmark, swapping the grading rule is enough to reorder the board. We think this is the most under-discussed fact in the field, because it means a vendor can be honest, reproducible and still irrelevant to your decision.

This page is narrower than our earlier post on reading agent benchmarks honestly, which covers agent evaluation in general. Here the subject is one specific confusion: why two text-to-SQL boards rank the same systems differently, and what that tells you about which number to trust. The head-to-head case, Cortex Analyst against Genie, has its own page; this one is about the machinery that makes any such comparison move. We use the vocabulary from our text-to-SQL evaluation taxonomy for the moving parts, specifically four of its dimensions: grading rule, answer shape, benchmark corpus and failure mode. The taxonomy is ours and it's vendor-neutral.

Take a hypothetical finance platform team picking a text-to-SQL layer for a Snowflake warehouse with a few thousand columns and fifteen years of accumulated naming conventions. They have two shortlisted vendors and two leaderboards that disagree. We'll come back to them at the end, because the honest answer to their question is not on either board.

Three benchmarks that measure three different things

Start with what the benchmarks were built to do, in their authors' words.

Spider 1.0 (EMNLP 2018) is a cross-domain schema-generalisation test. It is "a large-scale, complex and cross-domain semantic parsing and text-to-SQL dataset annotated by 11 college students," with 10,181 questions and 5,693 unique complex SQL queries across 200 databases covering 138 domains. The defining choice is the split: different databases appear in train and test, so "the task requires the model to generalize well to both new SQL queries and new database schemas." The databases themselves are small and tidy.

BIRD (NeurIPS 2023) was built because that tidiness was the problem. Its authors note that Spider and WikiSQL "focus on database schema with few rows of database contents leaving the gap between academic study and real-world applications." BIRD ships 12,751 text-to-SQL pairs over 95 databases totalling 33.4 GB across 37 professional domains, and adds three things Spider does not have: dirty database contents, external knowledge that bridges the question and the stored values, and SQL efficiency. At publication, the best model managed 40.08% execution accuracy against a human result of 92.96%.

Spider 2.0 (ICLR 2025) moved again, this time to enterprise workflows. Its 632 problems come from real data applications, with databases "often containing over 1,000 columns and stored in local or cloud database systems such as BigQuery and Snowflake." Solutions routinely run past 100 lines and span several queries. The same code agent that solved 91.2% of Spider 1.0 and 73.0% of BIRD with o1-preview solved 21.3% of Spider 2.0.

Table 1 — Three corpora, three design goals. Counts are from each benchmark's own paper.

Those are three different jobs. A system tuned for schema linking across many small unfamiliar databases is being asked for something else than a system that has to read dialect documentation and write a hundred lines of Snowflake SQL. Ranking disagreement between them is not a contradiction to be resolved. It is the benchmarks working.

The grading rule changes the ranking

Here is the part that surprises people. Even inside one benchmark, on one set of answers, the rule that converts a returned table into a verdict decides the order.

Four rules are in circulation. Exact set match compares the query text, not the result: Spider 1.0 decomposes each query into clauses and does set comparison within each clause, ignoring literal values. It punishes any correct query written differently from the reference. Execution accuracy runs both queries and compares results. Test-suite accuracy runs them against several generated databases, so that a wrong query cannot survive by coincidence on one. Containment, which we call got-the-facts, asks only whether every reference value appears somewhere in what came back.

The difference is not philosophical. It's a few lines of each project's evaluator. BIRD:

# bird-bench/mini_dev — evaluation/evaluation_ex.py
def calculate_ex(predicted_res, ground_truth_res):
    res = 0
    if set(predicted_res) == set(ground_truth_res):
        res = 1
    return res

Set equality over result rows. Add one helpful column and every tuple changes, so the answer is wrong. The Spider test-suite evaluator is explicit about the same constraint, while permuting column order before it decides:

# taoyds/test-suite-sql-eval — exec_eval.py
num_cols = len(result1[0])
# if the results do not have the same number of columns, they are different
if len(result2[0]) != num_cols:
    return False

Spider 2.0 inverts it. Its comparison walks the reference columns and asks whether each one turns up anywhere in the prediction:

# xlang-ai/Spider2 — spider2-lite/evaluation_suite/evaluate_utils.py
tolerance = 1e-2
for _, gold in enumerate(t_gold_list):
    if not any(vectors_match(gold, pred, ignore_order_=ignore_order)
               for pred in t_pred_list):
        score = 0

Extra columns are free. Numbers match within 0.01. Row order can be ignored per example.

So each rule rewards a different style of answer and punishes a different innocent one. Exact result match rewards tight answers that return only what was asked, and wrongly fails a query that is right but shaped differently: extra context columns, a different column order, a rounded average, a DISTINCT the annotator didn't write. Got-the-facts rewards wide answers, and wrongly passes a query that returns the right numbers buried in a result set nobody could read, or that got there through a join path that will break next quarter. Exact set match rewards queries that look like the reference and fails novel SQL that is semantically identical. In taxonomy terms these are the tsqlet:grading-rule terms exact-result-match and got-the-facts, and the thing they disagree about is the tsqlet:answer-shape dimension. The taxonomy's own note on right-data-wrong-shape calls it the main reason rankings flip between benchmarks, and the evaluator code above is why.

BIRD's maintainers reached the same conclusion from the inside. The Mini-Dev set ships a soft F1 score, "a more lenient metric that reduces the impact of column order and missing values in the tables produced by predicted SQL queries," alongside EX and a reward-based efficiency score. Three rules, one question set.

Where rankings flip

Our own numbers are the cleanest demonstration we can point at, and they come with a conflict of interest we should state first: Agentic Fabriq ran this comparison, including against its own system, so treat it as first-party. Each product ran on its own platform with its own semantic tooling, on the same questions, with the same external grader and three runs per cell where possible.

Table 2 — First-party comparison published by Agentic Fabriq, exact match / got-the-facts.

Look at the Spider 1.0 column. Under exact match, Genie leads Cortex Analyst by 7.2 points, which anyone would report as a win. Under got-the-facts the gap is 0.4 points, which is a tie. Nothing about either system changed between those two readings. The grader changed, and with it the answer to "which of these is better at Spider 1.0."

Figure 1 — The same pair of systems, reordered by the grader alone.

We read the collapse from 7.2 to 0.4 as mostly answer shape: under containment, results that differ from the reference only in which columns came along stop being wrong. That reading is ours, not a measured decomposition, and we have not isolated it per question.

The public Spider 1.0 board makes the same point without us. It publishes two leaderboards over one benchmark, execution with values and exact set match without values. The top submission, MiniSeek, scores 91.2 on the first and 81.5 on the second. Below it the two lists barely overlap: the execution board is a run of 2023 GPT-4 prompting methods, while the exact-match board is still led by 2022 fine-tuned parsers like Graphix-3B + PICARD and CatSQL + GraPPa. Part of that is which board people chose to submit to. Part of it is that the two rules reward different machinery.

The answer keys are broken too

There is a worse problem underneath the grading rule, and it is well documented. The reference answers themselves are frequently wrong.

Jin, Choi, Zhu and Kang audited two benchmarks and published the result at CIDR 2026 with an extended version on arXiv in January 2026. They classify annotation errors into four patterns: question-versus-SQL mismatch, question-versus-database mismatch from limited understanding of the data and schema, missing or wrong domain knowledge, and plain ambiguity in the question. Then they counted. In BIRD Mini-Dev, 263 of 498 unique examples contain at least one such error, a rate of 52.8%. In Spider 2.0-Snow, 76 of the 121 examples whose gold SQL is public are wrong, a rate of 62.8%, which the earlier CIDR version of the paper put at 66.1%. They chose Snow over Spider 2.0-lite for a pointed reason: "teams at leading technology companies benchmark their agents on it."

Then they corrected a 100-problem sample of the 1,534-problem BIRD Dev set, fixing 19 questions, 17 external knowledge annotations, 41 gold SQL queries and one schema, and re-ran all 16 reproducible open-source agents from the BIRD leaderboard. Rankings moved an average of five positions, with individual shifts from −9 to +9. CHESS and GenaSQL went from 7th and 6th to a tie for first at 81% execution accuracy. MAC-SQL climbed from 14th to 5th. Contextual-SQL, previously first, fell to 7th. OpenSearch-SQL fell from 2nd to 11th.

The authors also tested whether the uncorrected sample is at least a fair proxy for the full set. Agent rankings on the uncorrected subset track the full BIRD Dev set closely, Spearman's r = 0.85. Rankings on the corrected subset track it weakly, r = 0.32. Their conclusion is blunt: "rankings of methods are highly sensitive to the annotation errors in the benchmark."

This is not new, only bigger than previously thought. Wretblad and colleagues found in 2024 that noise in BIRD's questions and gold queries was prevalent and unevenly distributed across domains, and reported something that should stop anyone leaning on a leaderboard delta: "when evaluating models on corrected SQL queries, zero-shot baselines surpassed the performance of state-of-the-art prompting methods." The elaborate method's advantage was an artifact of the answer key.

False positives, false negatives, and why execution accuracy is not a reprieve

Execution accuracy is often sold as the honest alternative to exact match. It's better. It is not clean, and the sizes of its two error modes are measured.

A wrong query can return the right table by accident. Zhong, Yu and Klein built test-suite accuracy for exactly this, distilling small suites of randomly generated databases so a prediction has to survive more than one world. Checking results only on the single database shipped with Spider produces a false positive rate averaging 6.5% across 21 leaderboard submissions, rising to 11.0% on the extra-hard split and 17.6% in the worst case. Those are predictions counted correct that are not.

The reverse error is smaller but real. The adapted exact set match metric shows a false negative rate of 2.6% on average and 8.1% at worst, and on hard queries 4% on average and 12.1% at worst. These are the shape failures: a correct query marked wrong for column order, an extra column, a DISTINCT, a rounding difference. Spider 2.0's grader is lenient here on purpose, with its 0.01 numeric tolerance and optional row-order blindness.

And both error modes reorder boards. On the extra-hard split, the authors found four submissions each having more than three others ranked above them on single-database execution accuracy but below them on test-suite accuracy. On the hard split, five of 21 submissions each had four others scoring lower on semantic accuracy and higher on exact set match. That is the thesis of this page, measured by someone with no product to sell: the metric, not the model, produced the order.

What follows if you're choosing a system

Back to the hypothetical finance platform team with two vendors and two disagreeing boards. Nothing they can read will settle it, and here is the shape of why.

Figure 2 — Three of the four inputs to any leaderboard number belong to the benchmark, not to you.

Three of the four inputs are the benchmark's. Your schema has its own join paths and its own fifteen-year-old column names, your users have their own definition of an active customer, and your grading rule is whatever your analysts will accept in a Slack thread. So the practical move is to stop reading scores as rankings and start reading them as filters. A published number is good evidence of a floor and poor evidence of an ordering, which is enough to shortlist and never enough to choose.

Then measure. We wrote the method up separately in how to evaluate text-to-SQL on your own database: write the questions your team actually asks, decide in advance which grading rule matches what your users will accept, and check the answer key before you trust a score computed from it. Pick the rule first, because it is the choice that most determines the result and the one most often left to a default.

This applies to our own numbers. mnemiq leads every cell of Table 2, and we published it, and that is exactly the pair of facts that should make you discount it. The engine's own thesis is that accuracy does not transfer across databases, which is why a language model proposes SQL in it and deterministic checks on shape, access, dialect and query plan decide whether the query runs. A benchmark score cannot tell you whether those checks fit your warehouse. Only your warehouse can.

FAQ

Why does my system rank higher on Spider than on BIRD? Most likely because Spider 1.0's databases are small and clean while BIRD's contain dirty values and need external knowledge to connect a question to how the data is stored. BIRD was built specifically to close that gap. A drop from Spider to BIRD is the expected direction, not a regression.

Which text-to-SQL benchmark should I trust? For a shortlist, the one whose corpus most resembles your situation: Spider 2.0 if you run a large cloud warehouse with thousand-column tables, BIRD if your data is messy and your questions need business context. For a decision, none of them.

Is a two-point difference on a BIRD leaderboard meaningful? We don't think so. An audit found annotation errors in 52.8% of BIRD Mini-Dev examples, and re-scoring 16 agents on corrected data moved rankings by an average of five positions. A two-point gap is inside that noise.

Does Spider 2.0 replace BIRD? No. It tests a different thing, and it grades differently, accepting any result that contains the reference values. Scores from the two are not comparable in either direction.

Can I put a vendor's published score in a procurement document? You can cite it as a floor on capability if you also record the corpus, the grading rule, the model and the date. Presenting it as a ranking is the part that misleads, including when the vendor is us.

Conclusion

BIRD and Spider rankings disagree because they are measurements of different things taken with different instruments against answer keys that are themselves substantially wrong. Each of those three facts is documented, and each on its own is enough to reorder a leaderboard. Together they mean the sentence "system A beats system B at text-to-SQL" is incomplete to the point of being unfalsifiable.

What remains useful is the mechanism. Knowing that BIRD's grader fails an extra column and Spider 2.0's does not tells you something real about why two numbers differ, and it tells you which rule to pick when you grade your own. That is a transferable skill. A leaderboard position is not.

Sources

• Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task — Yu et al., EMNLP 2018. Source for the 10,181 questions, 5,693 unique SQL queries, 200 databases, 138 domains, and the cross-domain split design.
• Spider 1.0 leaderboard — the two parallel boards, the definitions of exact set match without values and execution with values, MiniSeek at 91.2 and 81.5, and the near-disjoint top eights.
• Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLs — Li et al., NeurIPS 2023. BIRD's 12,751 pairs, 95 databases, 33.4 GB, 37 domains, the dirty-values and external-knowledge framing, and 40.08% versus 92.96%.
• bird-bench/mini_dev — the 500-example Mini-Dev set across 11 databases, the soft F1 motivation quoted here, R-VES, and the calculate_ex set-equality implementation.
• BIRD benchmark site — current execution-accuracy leaderboard and the human-performance baseline.
• Spider 2.0: Evaluating Language Models on Real-World Enterprise Text-to-SQL Workflows — Lei et al., ICLR 2025. The 632 workflow problems, the 1,000+ column databases on BigQuery and Snowflake, and 21.3% against 91.2% and 73.0%.
• Spider 2.0 site and leaderboards — the 547-example Snow and Lite variants and the current standings.
• xlang-ai/Spider2 evaluation suite — the containment comparison quoted above, including the 1e-2 numeric tolerance and the optional row-order blindness.
• Semantic Evaluation for Text-to-SQL with Distilled Test Suites — Zhong, Yu and Klein, EMNLP 2020. Test-suite accuracy, the false positive rates for single-database execution accuracy, the exact-set-match false negative rates, and the rank inversions between the two.
• taoyds/test-suite-sql-eval — the reference implementation whose result_eq requires matching column counts.
• Pervasive Annotation Errors Break Text-to-SQL Benchmarks and Leaderboards — Jin, Choi, Zhu and Kang, January 2026. The 52.8% and 62.8% error rates, the 16 re-scored agents, the −9 to +9 rank changes, and the Spearman correlations of 0.85 and 0.32.
• Text-to-SQL Benchmarks are Broken: An In-Depth Analysis of Annotation Errors — the CIDR 2026 version of the same work, with the E1 to E4 error taxonomy and per-pattern distributions.
• Understanding the Effects of Noise in Text-to-SQL: An Examination of the BIRD-Bench Benchmark — Wretblad et al., ACL 2024. Earlier evidence of prevalent noise in BIRD, and zero-shot baselines overtaking state-of-the-art prompting once gold queries were corrected.
• Text-to-SQL evaluation taxonomy — ours, vendor-neutral. The grading rule, answer shape, benchmark corpus and failure mode dimensions used throughout, including the right-data-wrong-shape and reference-error terms.
• mnemiq benchmark comparison — the first-party numbers in Table 2, the exact match and got-the-facts definitions, and the methodology note on runs and grading.
• agenticfabriq/mnemiq — the open-source engine whose design assumes accuracy does not transfer across databases.