Open source · Release
/NEM-ik/ — the "m" is silent, as in mnemonic
/NEM-ik/ — the “m” is silent, as in mnemonic
Before you trust AI on your company’s data, you should be able to test it on your own database, see where it fails, and improve it.
Text-to-SQL vendors publish accuracy numbers, but those numbers do not transfer across databases. They tell you little about how a system will perform on your data — and when it underperforms, the black box gives you little ability to understand why or fix it.
We built mnemiq, an open-source text-to-SQL engine that opens up that black box. You can change the model, retrieval, semantic layer, verifier, and other parts of the pipeline, then benchmark those choices on your own database.
We tested mnemiq on BIRD mini-dev, Spider 1.0, and Spider 2.0-lite, ranging from relatively clean benchmark schemas to much larger, messier ones. Using the same external grader, mnemiq scored higher than Snowflake Cortex Analyst and Databricks Genie on all three:
mnemiq was also more stable across identical runs: its case-flip rate was 2.2–4.1%, compared with 9.4–17.3% for Genie and 10.6–12.8% for Cortex Analyst.
The comparison is useful, but it raised a better question: why did performance move so much when we changed the model, the context, and the data?
The more useful results came from underneath that comparison. Across 28 configurations, fixing a SQL dialect bug helped more than most model-level techniques, spending more compute often made answers worse, and semantic enrichment was decisive on one dataset and worth almost nothing on another.
Our point is simple: no text-to-SQL accuracy number applies to your database until you run it on your database.
mnemiq is the engine and the evaluation harness for doing that.
Every text-to-SQL product has an accuracy number. Almost none of them were measured on a database that looks like yours.
A system can perform well on a benchmark and then struggle on your warehouse because the schema is larger, the naming is different, the business definitions are different, or the configuration that worked on the benchmark simply does not work as well on your data.
That makes the question we actually care about:
How well will this work on my database — and what can I change if it doesn’t?
We built mnemiq to make both parts measurable. It is Apache-2.0, can run inside your own environment on models you choose, and exposes the major parts of the text-to-SQL pipeline as settings.
The outcome is not just another accuracy score. You can figure out which model works best on your data, whether semantic enrichment is worth maintaining, how much retrieval context you need, how aggressively the system should refuse uncertain answers, and whether paying for more model or more compute actually improves anything.
This post is the experiment log behind those decisions. We tested models, retrieval depths, semantic-layer settings, fine-tunes, verifiers, and decoding strategies, then compared mnemiq with Snowflake Cortex Analyst and Databricks Genie using the same public benchmarks and the same external grader. Across the experiments here, beacon graded more than 25,000 answers.
At a high level, mnemiq separates SQL generation from the decision to actually run it. The model proposes SQL, then a deterministic layer checks it before anything touches the database.
The query has to:
EXPLAINIf any of those checks fail, mnemiq will refuse the query and tell you why.
Permissions are applied before schema retrieval, so the model is never shown tables the user isn’t allowed to see. Every answer also carries a trace showing what SQL ran, what tables it touched, and what enrichment version it used.
Before questions are asked, mnemiq can inspect the database and build more context around the schema. There are 3 levels to this:
Tier 0: tables and columns only
Tier 1: adds structural information like primary keys, foreign keys, and profiling
Tier 2: adds semantic information generated by an LLM — including table descriptions, column descriptions, grain, glossary terms, and coded-value meanings.
For production use, semantic definitions can also be reviewed and certified by a human owner.
We tested the different tiers separately so we could figure out: How much does a semantic layer actually help?
It turns out that the answer to that depends strongly on the database.
mnemiq also lets you choose how much compute you want to spend on each question. The fast mode answers in one shot. The default mode gives a corrector up to three attempts to repair SQL if something fails. Deep mode generates five candidates and uses a judge and verifier to decide which answer to return, at roughly 6× the cost.
One of the more important things we found was that spending more compute did not always mean getting a better answer. In some cases it helped, and in others it actually made things worse. The verifier was the main exception: it was the only lever we tested that consistently reduced the wrong-answer rate.
That tradeoff is exposed as a setting because different applications have different risk tolerances. Some teams will want to answer as many questions as possible. Others would rather refuse more often if it means returning fewer wrong answers.
Before comparing mnemiq against Snowflake or Databricks, we wanted to see how much performance changed just by changing the engine itself.
So we ran 28 configurations on BIRD mini-dev, varying the model, retrieval depth, decoding strategy, fine-tuning, semantic layer, self-consistency, and verifier.
The result was more interesting than we expected: a local 14B model on a single RTX 4090 matched the hosted frontier model, while several techniques that sounded like obvious improvements actually made performance worse.
The chart below shows the tradeoff between accuracy and latency across all 28 configurations.
Most of the best speed/accuracy tradeoffs came from local open-weight models. In other words, for many points on the chart, there was no hosted configuration that was both faster and more accurate.
The most obvious outlier is the hosted model using five-sample self-consistency at around 34 seconds per question. It used roughly four times as much compute as asking the same model once, but scored about seven points worse.
That became a recurring theme: more model, more context, or more compute did not automatically mean better text-to-SQL.
One thing to keep in mind: this grid was part of the search process, not the final shipped configuration. Most rows change one setting at a time so we could isolate its effect. The configuration used later in the vendor comparison combines several of the settings that worked best, which is why its final BIRD score is higher than any individual row here.
We also publish the configurations that did not work. If we only showed the winners, you would not be able to tell which changes actually helped and which ones were dead ends.
A 14B local model matched the hosted frontier model. Qwen2.5-Coder 14B scored 50.9% exact match on BIRD in a single-shot run. The hosted frontier model’s best comparable grid result was 50.7%. The 14B model ran locally on a single RTX 4090 and answered in about 3.7 seconds per question. With five-sample voting, it reached 52.3%. We are not saying a 14B model is generally better than a frontier model. It obviously is not. What this showed us is that model size and model reputation alone are not enough to predict text-to-SQL performance. The rest of the system matters a lot.
The 32B model did not improve accuracy. Going from 7B to 14B helped a lot. Going from 14B to 32B did not. Qwen2.5-Coder 32B scored 50.3% with guided decoding, slightly below the 14B model at 50.9%, while requiring an A100 instead of a 24 GB consumer GPU. The 32B model did have one advantage: with the verifier turned on, it reached the lowest wrong-answer rate in the entire grid at 25.7%. But it got there by refusing about a quarter of the questions. That is why we think right, wrong, and refused need to be reported separately. A single accuracy number hides a lot of how the system actually behaves.
One of our biggest improvements came from fixing the SQL dialect. Around 35 logically correct queries were failing because the engine was generating SQL in one dialect and executing it against another. Fixing that improved the hosted frontier model by about seven exact-match points — a larger gain than most of the model-level techniques we tested. That was a useful reminder that a lot of text-to-SQL performance comes from things that look like plumbing until you measure them. It is also why mnemiq exposes those parts of the pipeline instead of hiding them behind one API call.
Accuracy and wrong-answer rate are not the same thing. Two systems can have the same exact-match score and still behave very differently. At exactly 33.9% exact match, the 7B model was wrong on 39.4% of questions. The 14B model, at the same exact-match score, was wrong on only 19.5%. The hosted frontier model’s best grid cell was wrong 48.2% of the time and refused only 1% of questions. In other words, it almost always attempted an answer, even when that answer was wrong. The verifier was the only setting we tested that consistently pushed the wrong-answer rate down. On the 14B model, it reduced wrong answers from 38.8% to 32.4%. On the 32B model, it got the wrong-answer rate down to 25.7%, but with much lower coverage. There is no universally correct point on that tradeoff. It depends on what you are building and how costly a wrong answer is.
| Model | Configuration | n | Exact | Defer | Wrong | Got-facts | ms |
|---|---|---|---|---|---|---|---|
| Qwen-14B | guided + self-consistency N=5 | 486 | 52.3% | 7.8% | 39.9% | 57.0% | 18,228 |
| Qwen-14B | guided, no semantic layer | 487 | 52.2% | 9.4% | 38.4% | 55.2% | 6,037 |
| Qwen-14B QLoRA-800 | baseline | 487 | 51.5% | 7.8% | 40.7% | 52.2% | 9,023 |
| Qwen-14B QLoRA | baseline | 487 | 51.1% | 10.3% | 38.6% | 52.4% | 8,937 |
| Qwen-14B | guided JSON | 487 | 50.9% | 10.3% | 38.8% | 54.0% | 3,719 |
| Frontier | retrieval k=12 | 483 | 50.7% | 1.0% | 48.2% | 63.4% | 8,427 |
| Qwen-14B QLoRA | guided JSON | 487 | 50.7% | 12.5% | 36.8% | 52.4% | 8,710 |
| Qwen-32B | guided JSON | 487 | 50.3% | 9.0% | 40.7% | 53.6% | 8,482 |
| Qwen-14B QLoRA-800 | guided JSON | 487 | 49.7% | 11.9% | 38.4% | 50.9% | 8,889 |
| Qwen-14B | verifier on | 487 | 48.5% | 19.1% | 32.4% | 52.2% | 7,120 |
| Frontier | guided JSON | 486 | 47.7% | 1.0% | 51.2% | 63.2% | 10,037 |
| Frontier | dialect fix, single-shot | 482 | 47.7% | 5.4% | 46.9% | 61.4% | 7,410 |
| Qwen-32B | verifier on | 486 | 46.1% | 28.2% | 25.7% | 49.2% | 10,403 |
| Frontier | facts-only enrichment | 483 | 44.9% | 4.6% | 50.5% | 61.3% | 8,686 |
| Qwen-14B | assertive prompt (fixed) | 487 | 44.8% | 17.5% | 37.8% | 47.8% | 3,433 |
| Frontier | self-consistency N=5 | 483 | 43.7% | 12.6% | 43.7% | 54.2% | 34,173 |
| Qwen-32B | baseline | 487 | 43.7% | 23.4% | 32.9% | 46.6% | 7,152 |
| Frontier | pre-dialect baseline | 482 | 40.7% | 13.5% | 45.9% | 53.5% | 8,004 |
| Frontier | pre-retrieval single-shot | 484 | 40.5% | 13.2% | 46.3% | 53.7% | 8,084 |
| Frontier | facts + examples | 481 | 40.3% | 4.2% | 55.5% | 60.7% | 8,682 |
| DeepSeek-V2-Lite | baseline | 487 | 37.6% | 18.1% | 44.4% | 41.9% | 9,842 |
| Qwen-14B | guided JSON (pre-fix) | 487 | 35.9% | 41.1% | 23.0% | 37.6% | 4,821 |
| Qwen-14B | assertive prompt | 487 | 33.9% | 46.6% | 19.5% | 35.5% | 3,620 |
| Qwen-7B | guided JSON | 487 | 33.9% | 26.7% | 39.4% | 35.1% | 3,277 |
| Qwen-7B QLoRA | guided JSON | 487 | 19.7% | 56.9% | 23.4% | 19.9% | 4,355 |
| Qwen-7B QLoRA | baseline | 487 | 19.3% | 62.2% | 18.5% | 19.5% | 3,217 |
| Qwen-14B | baseline | 487 | 17.0% | 73.9% | 9.0% | 17.7% | 5,578 |
| Qwen-7B | baseline | 487 | 6.4% | 92.0% | 1.6% | 6.6% | 1,367 |
The practical takeaway from this grid is simple: on BIRD, a local 14B model running on one RTX 4090 was competitive with the hosted frontier model through the same engine. But that result did not hold uniformly once we changed the data. That is exactly the point of making these settings measurable.
Spider 2.0-lite
BIRD and Spider 1.0 are useful benchmarks, but their schemas are relatively clean compared with what most enterprise teams actually deal with.
Spider 2.0 is much closer to the shape of a real warehouse. The schemas are larger, there are many more columns, and the system has to retrieve the right part of the schema before it can even start writing the SQL.
That is also where the gaps between systems became much larger.
| Enrichment | BIRD mini-dev | Spider 1.0 | Spider 2.0-lite |
|---|---|---|---|
| tier 2 semantic | 52.2 / 64.9 | 68.4 / 79.5 | 36.6 / 59.8 |
| tier 1 structural | 50.8 / 63.8 | 69.0 / 80.1 | 36.2 / 59.2 |
The numbers above are exact match / got-facts.
One of the more interesting results here is that the semantic layer did not help equally on every benchmark. On BIRD, it added around a point. On Spider 1.0, it basically did nothing. On Spider 2.0, it helped a little more.
The biggest improvement from semantic enrichment actually came from an in-domain dataset, not a public benchmark. When we gave the system certified business definitions, performance landed around 69–72%. Without those definitions, it was around 51.7–58.6%.
That makes sense when you look at what semantic enrichment is actually doing.
If your schema already contains most of the meaning the model needs, more descriptions may not help much. If you have columns like rev_amt, rev_amt_net, and rev_settled, and different teams casually call all three “revenue,” then the definition matters a lot.
That is why mnemiq does not assume everyone needs the same semantic layer. It gives you the setting and the evaluation harness so you can measure whether the extra work is actually worth it on your database.
A system can have a respectable average accuracy score and still be hard to trust in production if the same question gives you a different answer every time you ask it.
So we measured stability: ask the exact same questions in separate runs and count how often the verdict changes.
| System | case-flip rate between identical runs |
|---|---|
| mnemiq | 2.2–4.1% |
| Genie | 9.4–17.3% |
| Cortex Analyst | 10.6–12.8% |
mnemiq’s case-flip rate was 2.2–4.1%. Genie’s was 9.4–17.3%, and Cortex Analyst’s was 10.6–12.8%.
Put differently, roughly one in eight vendor answers changed between identical runs in some pairings. That matters operationally. A dashboard, analyst, or agent that gives a different answer on Tuesday and Thursday is difficult to trust even if its average benchmark score looks good.
We think mnemiq is more stable partly because most of the pipeline after SQL generation is deterministic. The model’s randomness is concentrated in fewer places.
Neither Snowflake nor Databricks currently publishes a stability number for these products. We think they should. dbt Labs already moves in this direction by repeatedly running questions in its semantic-layer benchmark and reporting which stay consistent.
We did not want the vendor comparison to be the main point of this article.
A benchmark run by one of the products being compared deserves skepticism, including ours. So if we were going to publish the comparison, we wanted the setup to be as inspectable as possible.
Each product ran on its own platform with its own semantic tooling. We used the same benchmark questions, the same external grader, and three runs per cell where possible. We also publish the per-question results so you can inspect the SQL and outputs rather than trusting the summary table.
| System | BIRD mini-dev | Spider 1.0 | Spider 2.0-lite |
|---|---|---|---|
| mnemiq (tier 2, GPT-5.5) | 52.2 / 64.9 | 68.4 / 79.5 | 36.6 / 59.8 |
| Databricks Genie | 37.4 / 53.1 | 63.6 / 77.5 | 27.3 / 45.5 |
| Snowflake Cortex Analyst | 33.3 / 48.9 | 56.4 / 77.1 | 17.4 / 24.2 |
Again, these are exact match / got-facts.
mnemiq scored higher on both metrics across all three benchmarks. The largest gap was on Spider 2.0-lite, where got-facts was 59.8% for mnemiq, 45.5% for Genie, and 24.2% for Cortex Analyst.
But we want to be careful about what that result does and does not prove.
The vendor models are undisclosed. Part of the difference may come from model choice. Part may come from the semantic setup. Part may come from architecture. And none of these benchmarks tells you exactly what will happen on your own warehouse.
So we see the vendor comparison as evidence for the larger point, not the point itself.
Cross-vendor benchmarks are easy to get wrong. We found that out several times while running this one.
Instead of quietly fixing problems and publishing the final table, we kept track of the issues we found and how we handled them.
Cortex Analyst refused 27 of the 135 Spider 2.0 questions. We manually read all 27 refusal messages. Twenty-four were caused by missing join metadata, two were genuine inability to answer, and none were clarification requests.
We still report the result as the product behaved under that setup, but we think the reason for the refusals matters. The benchmark score alone would not tell you that.
Every number in this article depends on what we decide counts as correct, so we also spent time testing the grader itself.
Partway through the project, we tightened the exact-match rule so column order became part of the score. Instead of mixing results from the old and new grading rules, we re-executed and regraded every stored prediction from every system. That reduced exact-match scores by between 1.4 and 5.7 points depending on the system. Got-facts barely moved.
That is one reason we put more weight on got-facts in the comparison. It was much less sensitive to changes in the grader.
As a second check, we wrote two separate implementations of the got-facts logic and ran them against 414 constructed examples. They agreed on all 414.
There are a few things worth knowing before reading too much into any individual number.
The full matrix compresses a lot of the experiment into short configuration names, so below is what each major setting actually changes and what happened when we tested it
CUST_STAT_CD='A'. We do not want the model guessing that A means “active.” mnemiq only attaches a meaning when it has evidence from the database, a lookup table, a dictionary, or another configured source. If two sources disagree, it leaves the value unresolved instead of picking one. This was mostly neutral on BIRD, but it removed a class of hallucinations that matters much more in real databases.baseline means the default mode with no extra settings enabled. guided_json is constrained decoding. assertive is the attempt-rather-than-refuse prompt. SC N=5 is five-sample self-consistency. verify means the verifier is on. k12 and k24 are retrieval depth. qlora and qlora800 are the fine-tuned adapters. dialect fix means source-native SQL generation and execution. single-shot means one candidate and one attempt. The local rows ran quantized open-weight models on an RTX 4090. The larger local setup used an A100.
Six of the fourteen major levers we measured were flat or negative. We left them in because failed experiments are part of the result too.
None of the numbers in this article tell you what will happen on your warehouse. That is not a disclaimer around the benchmark. It is the main finding.
We saw the best configuration change when we changed the data. Semantic enrichment mattered a lot in one environment and barely mattered in another. A local 14B model matched a hosted frontier model on BIRD, but that does not mean it will do the same thing on your schema.
So the goal is not to replace Snowflake’s benchmark with ours. The goal is to get your own number — and, more importantly, understand what is behind it.
You do not need a huge evaluation program. Take one meaningful slice of your schema and write 20–30 questions people actually ask. Tag each one as answerable from the schema, requiring a business definition, or something the system should refuse.
Then test a few meaningful choices: enrichment on and off, a local and hosted model, maybe two verifier thresholds. Run each setup at least twice and report right, wrong, and refused separately.
What you end up with is more useful than a vendor accuracy number. You know which setup performs best on your data. You know where it fails. You know whether semantic work is actually helping. You know how often it gives a wrong answer instead of refusing. And you know whether paying for a larger model or more compute is buying you anything.
If definition-heavy questions fail, you have evidence that better semantic documentation is worth the work. If everything works fine without enrichment, you have evidence that you do not need to build and maintain a heavy semantic layer. If a small local model performs just as well as a hosted one, you can make that infrastructure decision with data instead of guessing.
That is what mnemiq is for: not giving you another accuracy number, but giving you the tools to find the text-to-SQL system you can actually trust on your database.
This is not the end of our work on text-to-SQL. We are still measuring the parts of this system we have not measured yet, and there is more to come on databases, semantics, and the agents that read them.
Keep up with us on LinkedIn, on X, and here on this site for future launches.