How to Evaluate Text-to-SQL on Your Own Database

Overview

Someone has asked you to say whether a text-to-SQL system is good enough for your warehouse. The vendor has a deck with a BIRD score on it. You know, or suspect, that the score is not about your database.

That suspicion is correct and we've argued it separately: why BIRD and Spider rankings disagree takes apart the public corpora and the ordering flips between them. This is the other half, and it is the useful half. Assume no leaderboard transfers. What do you run to get a number about your own schema, and what has to be true of that number before you quote it?

The parts that decide the answer are boring and mostly not the model: the question set, the reference answers, the grading rule, the number of runs, and what happens to a question the system declines. Get those wrong and you still get a percentage. It just isn't a measurement of the thing you care about. Governance controls are a separate subject, covered in mnemiq and governed data agents; this is the measurement procedure, and it assumes you know which schema slice you may point at.

Figure 1 — The evaluation loop, with every decision point named. The two gates that get skipped are stratification and writing the grading rule down first, and skipping either makes the loop's output uninterpretable.

Where the Questions Come From, and How Many

Questions you invent at a whiteboard are biased toward the schema you remember. Real questions are biased toward what people need, which is the bias you want.

Three sources carry them. Your BI tool's query log is the richest: Looker, Tableau and Metabase all retain executed queries, and a quarter's worth tells you which tables carry traffic and which joins analysts actually write. The analytics team's backlog is second, and better in one respect, because a backlog item is a question nobody has answered yet. Third, support and internal-helpdesk tickets, where the phrasing is unpolished in the way a real user's phrasing is unpolished. "how many enterprise accounts churned last quarter" is a harder and more honest input than "SELECT COUNT(*) FROM accounts WHERE ...".

Then stratify, because uniform sampling buries the result. Take a renewals analytics slice at a B2B software company: a third of the questions are single-table counts, a third need a join through a lookup table, and a handful turn on what "churned" means. Sample uniformly and the easy third dominates the percentage.

The bands we'd use, which are mnemiq's own tags in its README guidance, are answerable from column names, needs a definition, and should be refused. The diagnosis falls out of the comparison. Definition-band failures with schema-band passes mean a glossary problem, and writing one will pay for itself. Schema-band failures mean retrieval or enrichment. Both bands passing while the should-refuse band answers confidently means the system has no brakes.

Be honest about what each count buys. Twenty to thirty questions is a diagnostic: enough to say which band is broken, not enough to rank two configurations. Ranking needs real statistical power, and the numbers are larger than people expect. Evan Miller's paper on eval statistics works the arithmetic for a paired comparison and lands on roughly 969 questions to detect a three-point difference at 80% power and 5% significance. Two of its five recommendations bite hardest here: clustered standard errors when questions arrive in related groups, and inference on question-level paired differences rather than on two independent means. Clustering is unavoidable in this domain, because ten questions about the same fact table share its quirks and are not ten independent draws. Pretending otherwise makes your confidence interval too narrow and your improvements too convincing.

Reference Answers, and Deciding What Correct Means

A reference answer is not the gold SQL. It is the rows the gold SQL returns, and that is the artifact grading should compare against. Grading on the SQL string punishes a different-but-equivalent query, and equivalent queries are the normal case.

Have a person who knows the data write the query, run it, and store the result. Then write down, before running anything, which of these rules decides whether a candidate matches. The text-to-SQL evaluation taxonomy calls this the grading rule dimension, and its four terms are the live options:

Decide first because the same run scores differently under each. The taxonomy's answer shape dimension names the band that moves: tsqlet:answer-shape/right-data-wrong-shape, defined as a result that "would pass got-the-facts but fail exact-result-match". An answer carrying region, revenue, order_count when the reference carries region, revenue is in that band. Exact match fails it. Got-the-facts passes it. Neither rule is wrong; they answer different questions, and an analyst who gets an extra column rarely files a bug.

Figure 2 — One answer, three rules, three verdicts. The rule is not a detail of the measurement; on this band it is the measurement.

Beacon's grading contract shows how specific this has to get before it is reproducible. Got-the-facts there tolerates extra candidate columns, column order, row order, a decimal rounding either way under either convention, and the same value spelled differently. It refuses a missing gold column, a different row count, a different value, and NULL versus 0 versus empty string. Whole numbers compare exactly under both readings, because "a count, a year, an id is right or wrong, never within a tolerance band." That is the detail a rule needs. "Correct" is not a rule.

And exact match isn't a safe default just because it is strict. The test-suite accuracy paper measured Spider's own exact-set-match metric against 21 leaderboard submissions and found a false-negative rate of 2.5% on average and 8.1% at worst, so a strict rule marks genuinely correct queries wrong often enough to reorder a leaderboard. Report both readings; the disagreement is information, not noise.

Run It More Than Once

One run is not a measurement, and the reason is mechanical rather than philosophical. Stability is its own dimension in the taxonomy, with tsqlet:stability/case-flip-rate defined as the share of questions whose graded verdict changes across repeated identical runs, measured by running the full set at least three times with identical configuration and counting items not graded identically every time.

Temperature zero does not save you. Thinking Machines' write-up on inference nondeterminism sent 1,000 identical requests to Qwen3-235B at temperature 0 and got 80 unique completions, every one identical for the first 102 tokens and diverging at token 103. Their diagnosis is batch invariance rather than floating-point luck: "the primary reason nearly all LLM inference endpoints are nondeterministic is that the load (and thus batch-size) nondeterministically varies!" Your batch size is someone else's traffic.

Local servers add a second trap: they sample unless told not to. mnemiq's README records measuring this on vLLM 0.9, where Qwen2.5-Coder-32B ships temperature 0.7, a request naming no temperature gets the model's default, and five identical requests came back five different ways. Pinning temperature is necessary and not sufficient.

Report three things. The mean over runs rather than the best run, which the taxonomy calls tsqlet:stability/mean-of-runs. The spread between best and worst. And the case-flip rate, the number a stakeholder actually needs: the share of questions on which the system has no settled answer. A system at 64% mean with a 3% flip rate and one at 64% with an 18% flip rate are not the same product, and a single percentage cannot tell them apart.

Three runs is the floor. Resampling also reduces variance, so if a cell matters enough to argue about, run it more and say how many times.

Count Refusals Separately From Errors

A system that says "this schema has no revenue column" and a system that returns a confident wrong number are not the same system, and one accuracy percentage cannot see the difference. Folding refusals into the denominator as failures makes a cautious configuration and an inaccurate one converge to the same score, which is how over-deferral becomes invisible.

Beacon states both commitments plainly. "A deferral is not a failure. A system that declines to answer scores DEFER, not FAIL." And "an error is not a zero. An attempt that never ran leaves the denominator instead of counting against the model. A rate over nothing gradeable is None, not zero."

So you need five buckets: correct, wrong, refused-correctly, refused-wrongly, errored. The taxonomy's refusal dimension names the rates. tsqlet:refusal/deferral-rate is the share declined. tsqlet:refusal/over-refusal is a question it declined but could have answered, which is the cost of caution. tsqlet:refusal/calibrated-refusal is the one worth chasing: the questions a system declines are ones it would more often get wrong if forced, measured by comparing a lower-deferral configuration's accuracy on the items the higher-deferral configuration declined against its overall accuracy.

That measurement is the only way to tell a working brake from a broken one, and it needs two configurations. Run the set with the verifier threshold high, then low. If the questions the strict configuration refused are ones the permissive configuration got wrong, the refusals earn their keep. If the permissive configuration answered them correctly, you're paying for caution in lost coverage, and the report should say so.

The should-refuse band gives you the other half: deliberately unanswerable or unauthorized questions, where answering is the failure.

Who or What Grades It

Three options, and each fails in a specific way.

Result-set comparison is the default and should be. It compares the rows your engine returned against the reference rows, under a rule you wrote down, and it is reproducible, cheap and auditable. It fails on questions whose answer is not a result set, and on reference errors: the taxonomy lists tsqlet:failure-mode/reference-error for a gold query that is itself wrong, which happens often enough that a strict rule will sometimes mark the model right and your reference wrong.

Human review is ground truth and does not scale. It also drifts, because a reviewer on question 200 is not the reviewer who did question 5. Use it on a stratified sample, with the acceptance criteria written first and two people grading an overlap so you can quote their agreement rate.

An LLM grader is the tempting middle, and the honest version is more work than people expect. The MT-Bench paper found strong judges reaching "over 80% agreement" with human preferences, "the same level of agreement between humans", which is the argument for using one. The same paper names position bias, verbosity bias and self-enhancement bias, which are the arguments against trusting one you haven't characterized. The taxonomy's caution on tsqlet:grading-rule/llm-judged is one line and it is the whole problem: "Not reproducible across judge models; report the judge and prompt."

Reproducible means: publish the exact prompt, pin the judge model and version, pin temperature at zero, publish per-question verdicts rather than an aggregate, and report the judge's agreement with human review on a sample of a hundred or so items. Without that last number you've replaced a measurement with an opinion that scales.

Never let the grader be the model that wrote the SQL. Panickssery, Bowman and Feng found LLM evaluators recognize their own outputs at "non-trivial accuracy" without special training, score those outputs higher while human annotators rate them equal, and that the bias strength tracks self-recognition ability linearly. Beacon handles it by exclusion: its LLM judge "serves narrative and rubric graders only — never the SQL path."

A Worked Example You Can Run Today

mnemiq is our open-source text-to-SQL engine, so this section is first-party and the pipeline is readable in the repo. It is here because it ships the three things this method needs and most systems don't expose: an open grader, a tracker holding every run, and per-question results.

Start with the demo, which needs no database of your own and no Docker. The seed step writes a small SQLite database plus its source manifest and access policy under demo/.

uv sync
uv run python scripts/seed_demo.py

export MNEMIQ_LLM_BASE_URL=...  MNEMIQ_LLM_API_KEY=...  MNEMIQ_LLM_MODEL=...
export MNEMIQ_SOURCES_PATH=demo/sources.json
export MNEMIQ_AUTHZ_PATH=demo/authz.json
export MNEMIQ_STORE_PATH=demo/store.duckdb

uv run mnemiq enrich      # profile + describe the schema
uv run mnemiq build       # index it for retrieval
uv run mnemiq ask "how many customers are there by country?" --roles analyst

--roles analyst is not optional. Access is fail-closed, so without a role the engine grants nothing and defers, which the README notes is "the first thing people mistake for a bug". Two more questions show the parts that aren't the model:

uv run mnemiq ask "how many enterprise customers are there?" --roles analyst
uv run mnemiq ask "what was our total revenue last quarter?"   --roles analyst

The first has to join through a lookup table to resolve a coded column, because segment_cd holds A/B/C and nothing in the name says "enterprise". The second is refused: the demo schema has no revenue column, and the engine says so instead of returning a number. Definition band and should-refuse band, in two commands.

Your own question set is a JSON file of cases, in the shape of the shipped evals/acme.json:

[
  {"id": "churn-q1", "question": "how many enterprise accounts churned last quarter?",
   "gold_sql": "SELECT count(*) AS n FROM account a JOIN segment s ON s.cd = a.segment_cd WHERE s.label = 'enterprise' AND a.churned_at >= '2026-07-01'",
   "tags": ["aggregate", "join", "coded-value", "needs-definition"]}
]

Then grade it. The harness is part of the engine rather than a separate research project, and it reports six outcomes: CORRECT (exact result-set match), CORRECT_FACTS (right data, different shape), WRONG, DEFERRED_CORRECTLY, DEFERRED_WRONGLY, ERROR. It prints two accuracies from one run, got-the-facts over answerable and exact match over answerable, which is the two-readings discipline enforced in code.

# the golden set, with a regression gate and a recorded trend point
uv run mnemiq eval --golden evals/mine.json --gate --record

# enrichment on, then off, with per-case artifacts: question, gold SQL,
# engine SQL, both result sets, outcome
uv run python scripts/run_eval.py --ab --report reports/

# a fixed failure becomes a permanent case
uv run mnemiq feedback --question "how many enterprise accounts churned last quarter?" \
  --sql "SELECT ..." --tables account,segment --golden evals/mine.json

One wrinkle, because it will bite you. mnemiq eval reads its golden set through the Postgres adapter whatever the source manifest says, so it wants MNEMIQ_PG_DSN and refuses on a SQLite-only deployment. The no-Docker demo is for the ask loop and the refusal behaviour; point the eval at a Postgres slice of your own.

For third-party grading, push the outputs to beacon, a tracker and not an executor: you register a run, execute it on your own hardware with your own runner, and push the rows your engine returned. Beacon grades those rows against gold it already holds, so no benchmark database is needed at grading time and the system under test never marks its own homework. beacon suites create makes the question set, beacon eval run registers the run, scripts/load_eval_reports.py loads JSONL reports, and beacon audit spot-check re-executes a sample of the pushed SQL on a named engine. Loading a report doubles as a conformance check between two independent graders, and the disagreement table it prints is the reason to bother.

What Makes a Number Dishonest

Four ways, and all four are easy to do by accident.

Tuning on the test set. You run the set, see the definition band fail, add the definitions, rerun, report the new number. Good engineering loop, ruined measurement, because the set is now a training signal. Keep a held-out slice you only touch when you report. If you have 30 questions and can't afford to hold any out, report the tuned number and label it as tuned.

Grading with the model that wrote the SQL. Covered above: self-recognition tracks self-preference, and the bias is measurable. Different family for the judge, or no LLM judge on the SQL path at all.

Reporting one run. Eighty unique completions out of a thousand identical temperature-zero requests is what the floor looks like. A single run has no error bar and you can't add one afterwards.

Quoting a public benchmark number as yours. A BIRD score is a statement about databases somebody else curated. Repeating it in a slide about your warehouse is the move this method exists to replace, and it is the subject of the companion piece on why BIRD and Spider rankings disagree.

A fifth, quieter one: changing the grader and comparing across the change. Beacon's re-basing note is the honest version. Spider EX quoted before 2026-08-07 was 53.4% on a local slice under the runner's pre-convergence grading, and 55.7% on the same run once EX derived from beacon's verdicts. Their rule is that comparisons across that boundary must re-base, with the run config and dataset version distinguishing the two eras. Change the grader and every earlier number is a different metric.

Where mnemiq Fits, Honestly

mnemiq is ours, it is Apache-2.0, and the published comparison on /mnemiq is first-party: Agentic Fabriq ran it, including against its own system. Read it with that in front of you.

Every cell is exact match / got-the-facts, three runs per cell where possible, same questions and the same external grader for every system.

Table 1 — The first-party published comparison. The vendor models are undisclosed, and none of these three benchmarks predicts what happens on your warehouse. Both caveats are load-bearing.

What makes it checkable despite being first-party is that the grader, the tracker and the per-question results are open, in a separate repository from the engine, with the grading contract written down and pinned by SHA-256 in both repos. A number traces to the SQL and the rows that produced it. That is the standard we'd hold any vendor to, including us, and it is also the reason to prefer a measurement you ran yourself over any of those cells.

Frequently Asked Questions

How do I evaluate text-to-SQL on my own database? Build 20 to 30 real questions from your BI query log or analytics backlog, tag each as answerable from column names, needs a definition, or should be refused, write reference answers by running a human-written gold query and storing its rows, fix the grading rule in writing before you look at any output, run the full set at least three times with identical configuration, and report right, refused and wrong separately plus the case-flip rate.

How many questions do I need? Twenty to thirty to diagnose which layer is broken. Several hundred to rank two configurations: Miller's worked example puts a three-point difference at 80% power and 5% significance at roughly 969 paired questions.

Should I use exact match or result containment? Both, side by side, with the gap between them named. Exact match is leaderboard-comparable and produces false negatives: the test-suite accuracy paper measured 2.5% average and 8.1% worst-case false negatives for Spider's own metric across 21 submissions. Result containment matches what an analyst accepts.

Can I use an LLM to grade text-to-SQL? For narrative or rubric answers, with the prompt published, the judge model and version pinned, temperature at zero, per-question verdicts published and an agreement rate against human review on a sample. Not for comparing result sets, where deterministic comparison is cheaper and reproducible, and never using the model that generated the SQL.

How many runs should I do? Three minimum, with mean and spread both reported. Temperature zero does not make a run reproducible: 1,000 identical requests to Qwen3-235B at temperature 0 produced 80 unique completions.

Do refusals count as wrong answers? No, and folding them in is the most common way an evaluation becomes useless. Refusals get their own rate, split into correct refusals and over-refusals. Errors that never produced a query stay in the denominator rather than counting against the model.

What is a good text-to-SQL accuracy number? There isn't one. The number is a joint property of your schema, your question mix and your grading rule. What you can compare is one question set under two configurations of the same system, and that comparison tells you what to build next.

How do I know if the evaluation itself is wrong? Have a second grader re-grade the same stored rows and read the disagreements. Beacon's report loader prints exactly that table.

Conclusion

The number you want is not hard to compute. It is hard to make honest, and almost all of that difficulty sits in decisions made before any SQL is generated: which questions, written by whom, stratified how, graded under which rule, averaged over how many runs, with refusals counted where.

For the renewals slice at that B2B company, a defensible first result looks like this. Twenty-eight questions from the BI log in three bands, reference rows written by the analytics lead, exact match and got-the-facts both reported, three runs, and five numbers rather than one: band accuracy, deferral rate, over-refusals, errors, case-flip rate. That takes an afternoon, and it tells you whether to spend the next quarter writing a glossary or tuning retrieval. A BIRD score tells you neither.

Sources

All URLs read 2026-10-12.

• mnemiq on GitHub — README: the evaluation harness as part of the engine, the 20–30 question first pass and its three tags, the SQLite demo quickstart with no Docker, enrich/build/ask --roles, fail-closed access, the coded-value and refusal demo questions, and the vLLM 0.9 temperature measurement.
• mnemiq src/mnemiq/eval/harness.py — the six outcomes: CORRECT, CORRECT_FACTS, WRONG, DEFERRED_CORRECTLY, DEFERRED_WRONGLY, ERROR.
• mnemiq src/mnemiq/eval/report.py — accuracy as got-the-facts over answerable, strict accuracy as exact match over answerable, and why portability enters one and not the other.
• mnemiq src/mnemiq/cli.py — the verified flags for eval --golden/--gate/--record, feedback, ask --roles/--json/--mode, and the note that eval reads its golden set through the Postgres adapter.
• mnemiq evals/acme.json — the question-set file format: id, question, gold_sql, tags.
• mnemiq scripts/run_eval.py — --ab and --report DIR, and what the per-case artifacts contain.
• beacon on GitHub — tracker not executor, the register/execute/push/read flow, deferral is not a failure, an error is not a zero, SQL is evidence never scored, the CLI surface including audit spot-check, and the LLM judge restricted to narrative and rubric graders.
• The beacon grading contract — EX / exact / got-facts, what got-facts tolerates and refuses, whole numbers compared exactly, the 5e-7/1e-5 bounds, and the 53.4% → 55.7% re-basing note.
• Text-to-SQL evaluation taxonomy — the eight dimensions, including grading rule, answer shape, refusal and stability.
• Taxonomy: grading-rule.yaml — the four rules, what each rewards, and the caution on LLM-judged grading.
• Taxonomy: answer-shape.yaml — right-data-wrong-shape and why rankings flip on it.
• Taxonomy: refusal.yaml — deferral rate, correct refusal, over-refusal, adversarial refusal hold, and how to measure calibrated refusal.
• Taxonomy: stability.yaml — case-flip rate and its three-run measurement protocol, mean-of-runs, per-schema spread.
• Taxonomy: failure-mode.yaml — reference error, plausible wrong number, and the rest of the diagnosis vocabulary.
• Adding Error Bars to Evals (Miller, 2024) — the five recommendations, clustered standard errors for grouped questions, resampling for variance reduction, and the ~969-question power example.
• Semantic Evaluation for Text-to-SQL with Distilled Test Suites (Zhong, Yu, Klein, EMNLP 2020) — 2.5% average and 8.1% worst-case false negatives for Spider's metric across 21 leaderboard submissions.
• Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (Zheng et al., 2023) — over 80% judge–human agreement, and position, verbosity and self-enhancement bias.
• LLM Evaluators Recognize and Favor Their Own Generations (Panickssery, Bowman, Feng, 2024) — self-recognition without training, self-preference where humans rate outputs equal, and the linear correlation between the two.
• Defeating Nondeterminism in LLM Inference (Thinking Machines, 2025) — 80 unique completions from 1,000 identical temperature-0 requests to Qwen3-235B, divergence at token 103, and batch invariance as the cause.
• mnemiq product page — the first-party published comparison, the grading rules beside the figures, and the three-runs-per-cell methodology note.