Snowflake Cortex Analyst vs Databricks Genie: Accuracy, With the Rule Attached

TL;DR

On our benchmark Databricks Genie scores higher than Snowflake Cortex Analyst on all three corpora, but on Spider 1.0 that lead is 7.2 points under one grading rule and 0.4 points under another. Same answers, same runs, two defensible definitions of correct, two different stories.

Read the conflict of interest first. Agentic Fabriq ran this comparison, and the third system in the table, mnemiq, is ours and scores highest. We are not a neutral party. What we can offer instead of neutrality is a grader you can read, per-question results you can inspect, and disclosure of every place the harness constrained a vendor.

The usual way these comparisons get published is one number per product. That hides the thing a buyer needs: Cortex Analyst's Spider 1.0 answers contain the reference facts about as often as Genie's and fail the strict row-and-column test far more often, while on Spider 2.0 it falls behind under both rules, and 27 of its 135 answers there were refusals.

We are not claiming the gap is only cosmetic. On Spider 2.0 it is 27.3 against 17.4 under exact match and 45.5 against 24.2 under got-the-facts, which is not a formatting difference.

If you are choosing between these two products, take the mechanics below and ignore the ranking. Then run twenty questions from your own warehouse through both.

Overview

Start with the disclosure, because everything below depends on how much you trust it.

Agentic Fabriq built and published this comparison. We also build mnemiq, the open-source text-to-SQL engine that appears in the same table and scores higher than both commercial products on every cell. A vendor publishing a benchmark it wins is the weakest form of evidence there is, and we would treat someone else's version of this page with suspicion.

So the useful question is not whether to believe us. It is which parts of this are checkable. The grader is open source and so is the tracker the numbers come from. The per-question results are published, including the SQL each system generated, so a disagreement can be settled by reading a query rather than by arguing about methodology. And where our own harness hurt a vendor, we say so: we found a type mismatch in our Databricks load that dropped around twenty foreign keys, fixed it, reran Genie, and used the better number.

The rest of this is about the two commercial products, how each one gets the business context it needs before it can answer well, and what the scores do and don't support. The companion posts cover the method rather than the ranking: how to evaluate text-to-SQL on your own database for running this yourself, and why BIRD and Spider rankings disagree for why the choice of corpus and rule reorders leaderboards.

The Numbers, and the Rule Attached

Every cell below reads exact match / got-the-facts. Exact match requires the same rows and the same columns as the reference answer. Got-the-facts counts an answer that contains the reference result, so a correct value wrapped in two extra columns passes the second rule and fails the first. Both are defensible, and they are not interchangeable.

The corpora are BIRD mini-dev, Spider 1.0, and a 135-question slice of Spider 2.0-lite. Each product ran on its own platform with its own semantic tooling, on the same questions, graded by the same external grader (beacon v7), with three runs per cell where possible.

Table 1 — First-party results published by Agentic Fabriq at /mnemiq. Exact match / got-the-facts, three runs per cell where possible. mnemiq is our own system.

Two things about this table that a summary row would bury.

The first is stability. We asked the same questions in separate runs and counted how often the verdict flipped. mnemiq's flip rate was 2.2 to 4.1 percent. Genie's was 9.4 to 17.3 percent and Cortex Analyst's was 10.6 to 12.8 percent. A two-point difference between two products with a twelve-point flip rate is not a difference. Neither vendor publishes a stability number for these products, and we think both should.

The second is that our own lead shrinks under scrutiny in exactly the way the vendors' gap does. Under got-the-facts on Spider 1.0, mnemiq is 2.0 points ahead of Genie. That is a cell where we would not claim to have beaten anyone.

The Rule Moves the Gap, Except Where It Doesn't

Here is the pattern worth the page. Take the distance between the two vendors under each rule:

Table 2 — Genie minus Cortex Analyst, in points, by grading rule. Derived from Table 1.

On Spider 1.0 the two products are effectively tied on whether the right facts came back, and seven points apart on whether the result had the reference shape. On BIRD the rule barely matters. On Spider 2.0 the choice of rule doubles the gap.

The clearest signal is each system's own spread between its two scores, a rough measure of how often it found the facts and missed the shape. Cortex Analyst's spread on Spider 1.0 is 20.7 points, the widest in the table. Its spread on Spider 2.0 is 6.8 points, the narrowest in the table. Genie's spreads sit between 13.9 and 18.2 across all three corpora, so its answer shape is roughly as reference-like everywhere.

Read together, those two numbers say something fairly specific. On the easiest corpus, Cortex Analyst's deficit is mostly about the form of the result rather than the facts in it. On the hardest corpus, shape has almost nothing to do with it, because the facts are not arriving at all.

We have one piece of direct evidence for why. On Spider 2.0, Cortex Analyst refused 27 of the 135 questions. We read all 27 refusal messages by hand: 24 were caused by missing join metadata, two were a genuine inability to answer, and none were requests for clarification. A refusal fails both rules, which is why the Spider 2.0 column looks the way it does.

The generalisation cuts both ways. The honest version is that these two products are closer than the headline numbers suggest on clean, small, single-database schemas, and further apart on large enterprise-shaped ones, and that part of the Spider 2.0 distance is a statement about how a benchmark loads into each product's semantic layer rather than about SQL generation. The taxonomy dimension for this is answer shape, which asks "How does the result's form differ from the reference?", alongside grading rule and refusal.

passes exact match AND got-the-facts | no +-- contains the reference result? -- yes --> fails exact match, passes got-the-facts no --> fails both refusal ------------------------------------> fails both -->

Figure 1 — One answer, two verdicts. The middle branch is where Cortex Analyst's Spider 1.0 answers mostly land, and the bottom branch is where its Spider 2.0 answers go.

Semantic Context Is the Real Difference

Score aside, this is what a buyer should be comparing. Neither product is a model with a schema pasted into the prompt. Both are systems that answer well only after someone describes the business, and they ask for that description in different forms.

Snowflake's unit is the semantic view, a schema-level object that defines "logical tables," dimensions, facts, metrics and relationships. The docs are direct about why it exists: "Generic AI solutions often struggle with text-to-SQL conversions when given only a database schema, as schemas lack critical knowledge like business process definitions and metrics handling." Relationships are part of that object, and the documented benefit is that "Join paths are predefined, ensuring correct multi-table queries." You author a semantic view with SQL DDL, through Semantic Studio, by importing YAML or Tableau and Power BI files, or with Autopilot, "an AI-assisted wizard in Snowsight." Legacy YAML on a stage still works but is no longer the recommendation.

Databricks starts from the catalog instead. A Genie Agent, the object its docs index still glosses as "formerly Genie Spaces", is "a domain-specific natural-language chat interface in Databricks where users ask questions of their data" over datasets registered in Unity Catalog. Crucially, "Primary and foreign keys defined in your schema are automatically saved as join relationships in the Genie Agent." On top of that you add example SQL queries, SQL expressions, plain-text general instructions, and trusted assets, which are "example SQL queries and SQL functions that provide verified answers to questions you anticipate from users." When one is used, the answer is marked verified.

The limits are worth knowing before you plan a rollout. A Genie Agent counts "Instructions (100 per agent)", where "each example SQL query, each SQL function, and the entire General instructions text block each count as one instruction", plus "a maximum of 200 knowledge store snippets" covering table descriptions, join relationships and SQL expressions. On table count, Databricks says to "Aim for five or fewer tables" and to "Work within the 50 table limit", with the advice to "prejoin related tables into views or metric views" beyond that.

SEMANTIC VIEW (logical tables, facts, metrics, relationships) ---> Cortex Analyst authored by Autopilot / SQL DDL / YAML import; verified queries attach to the same object

Databricks: Unity Catalog tables ---> GENIE AGENT ---> Genie declared PK/FK auto-saved as join relationships plus instructions, example SQL, trusted assets (100 instruction limit, 200 snippets) -->

Figure 2 — Where each product's business context comes from. Snowflake asks for an authored object; Databricks reads the catalog and asks you to annotate it.

That difference explains a practical failure we hit. Benchmark formats often lose primary and foreign keys when they are loaded into a warehouse, and both products lean on declared keys, so we replayed the keys from the benchmark metadata onto both platforms. Even then, 73 relationships could not be represented in Unity Catalog because of how it handles foreign keys and uniqueness constraints, and we disclosed those rather than working around them in a way nobody could reproduce. One Spider 2.0 schema has 38 tables, which exceeded the 30-table space limit in force when we ran, so we kept the largest connected component of the foreign-key graph. The documented limit is now 50, so that particular constraint has already moved.

Both are harness facts, not product verdicts, and they are why the Spider 2.0 column deserves the least weight of the three.

What Each Vendor Claims Itself

Neither vendor reports results on our setup, and their own published numbers were produced differently, so these belong next to ours rather than against them.

Snowflake's engineering blog, dated 29 August 2024, claims "90%+ SQL accuracy on real-world use cases" for Cortex Analyst, measured on an internal "benchmark suite of 150 questions that mirror the real-world tasks business users encounter, covering topics such as sales, marketing, finance, etc." The same post reports being "nearly 2x as accurate as single-prompt SQL generation from GPT-4o" and "about 14% more accurate than other solutions on the market." It also argues that public benchmarks mislead: on Snowflake's internal set, GPT-4o's "accuracy plummeted to 51%." We cannot reproduce any of that, because the question set is not public. The argument underneath it is one we agree with and make ourselves.

Databricks' comparable number is not a Genie number. A July 2025 post reports "73.5%" test accuracy on BIRD's "single-model, single-generation track" against the private test set, from a Qwen 2.5 32B Coder Instruct base trained with the company's TAO and RLVR stacks. The post is careful that this is not the product: the techniques are "the same techniques we're using to improve popular Databricks products like AI/BI Genie and Assistant", while "BIRD neither fully captures the real-world complexity of this task nor the full-breadth of real products like Databricks AI/BI Genie." Reading 73.5 as Genie's BIRD score would be a mistake, and it is not what Databricks wrote.

The two vendors' own in-product evaluation tools make the grading-rule point better than we can. Databricks benchmarks mark a question Correct only "if Genie's query result exactly matches the benchmark's query result", which is an exact-match rule chosen by the vendor. Snowflake's evaluations instead work "by executing the SQL generated by Cortex Analyst and comparing the results against your verified queries", with judges that "use Cortex LLM inference" and a correctness metric you select by version, where the UI "lists v1, v2, and v3". One vendor grades by result equality and the other by model judgment, under a rule that has three versions. Their own quality dashboards are not comparable with each other, let alone with ours.

One more thing a buyer should know, from Snowflake's own docs rather than from us. The Cortex Analyst pages now open with "Snowflake recommends transitioning to Cortex Agents, which supports every Cortex Analyst capability with higher answer quality." We benchmarked Cortex Analyst. Snowflake is steering new work elsewhere.

What This Comparison Doesn't Tell You

Four limits, in rough order of how much they should change your decision.

Neither vendor's model is pinnable. Snowflake's docs now publish a preference order, naming Claude Sonnet 4.6, Claude Sonnet 4.5, GPT 4.1, Arctic Text2SQL R1.5 and a Mistral Large 2 plus Llama 3.1 70b combination, but also state "You cannot choose a model directly" and that selection depends on region, cross-region inference configuration and model-level access control, and "may change from time to time." The Databricks docs we read do not name Genie's model at all. mnemiq's model is stated, which makes our cell reproducible and theirs approximate. Some unknown part of every gap in Table 1 is model choice, and this design cannot separate it out.

The corpora are not your warehouse. BIRD is "over 12,751 unique question-SQL pairs, 95 big databases with a total size of 33.4 GB" across "37 professional domains", and its mini-dev slice is 500 examples. Spider 1.0 is "10,181 questions and 5,693 unique complex SQL queries on 200 databases"; its official metrics are execution accuracy and exact set match without values, and its authors stopped accepting submissions in February 2024 after releasing the test set. Spider 2.0 is "632 real-world text-to-SQL workflow problems derived from enterprise-level database use cases", with databases that often exceed a thousand columns. None of them contains your column naming, your half-documented fact table or your three competing definitions of revenue. BIRD's own site puts human performance on its full dev set at 92.96 percent execution accuracy, so these are hard questions for people too.

The mnemiq column is not a like-for-like product comparison. mnemiq ran with tier 2 enrichment, which inspects the database and generates descriptions, glossary terms and grain before questions arrive. We configured it. We did not configure the semantic view or the Genie Agent with the care a real customer would after six months of use, because the benchmark gives nobody that history.

And a single number hides behaviour. Our own grid had two configurations at the same exact-match score with wrong-answer rates of 39.4 and 19.5 percent. A product that refuses a fifth of its questions is a different purchase from one that guesses.

Frequently Asked Questions

Is Snowflake Cortex Analyst or Databricks Genie more accurate? On our benchmark, Genie scored higher on all three corpora: 37.4 against 33.3 on BIRD, 63.6 against 56.4 on Spider 1.0 and 27.3 against 17.4 on Spider 2.0 under exact match. Under got-the-facts the Spider 1.0 result is 77.5 against 77.1, which is a tie. We ran this comparison and we sell a competing system, so weigh it accordingly.

Why do the two products look tied on one benchmark and far apart on another? Because the grading rule and the corpus both matter. On Spider 1.0 both products return the reference facts about as often, and Cortex Analyst's results more often fail the strict row-and-column test. On Spider 2.0 Cortex Analyst refused 27 of 135 questions, and a refusal fails every rule.

What accuracy does Snowflake claim for Cortex Analyst? "90%+ SQL accuracy on real-world use cases", from a 2024 engineering blog post, measured on an internal 150-question suite that is not public. It is not comparable with any number here.

What accuracy does Databricks claim for Genie? No product-level accuracy figure that we could find. Databricks reports 73.5 percent on BIRD's single-model, single-generation track for a research model, and says explicitly that BIRD does not capture the full breadth of Genie.

Does Cortex Analyst need a semantic model? Yes. Semantic views are the documented mechanism, with legacy YAML on a stage supported for backward compatibility. Autopilot can generate a first draft.

How many tables can a Genie Agent cover? Databricks documents a 50-table limit and advises aiming for five or fewer, with prejoining into views beyond that. The limit was 30 when we ran this benchmark.

Do declared primary and foreign keys matter? Enough to change a score. Genie saves schema PK and FK definitions as join relationships automatically, and Cortex Analyst reasons about joins from relationships declared in the semantic view. We replayed keys from benchmark metadata onto both platforms, and 24 of Cortex Analyst's 27 Spider 2.0 refusals still traced to missing join metadata.

Can I reproduce these numbers? The grader and tracker are at github.com/agenticfabriq/beacon, mnemiq is at github.com/agenticfabriq/mnemiq, and the per-question results are published with the comparison. The vendor runs depend on account access and on model selection we cannot pin, so expect to reproduce the method rather than the exact cells.

Conclusion

If you want one sentence: on our measurement Genie is ahead of Cortex Analyst, by 0.4 points on the easiest corpus under the looser rule and by 9.9 or 21.3 points on the hardest, depending on which rule you pick. That is two findings, not one, and quoting only the second is how vendor benchmarks get written.

The mechanics are more durable than the ranking. Cortex Analyst wants an authored semantic view with relationships declared in it. Genie reads Unity Catalog, picks up your declared keys as join relationships, and asks you to annotate within a hundred instructions and two hundred knowledge snippets. Whichever you pick, the work is the same work, and it is the part that decides your accuracy rather than the model underneath.

So treat this page as a method you can borrow and a set of numbers you should verify. The scores came from a grader we wrote, on corpora that are not your data, against vendor systems whose models we cannot pin, run by the company that makes the third system in the table. Every one of those is a reason to go measure on your own warehouse.

Sources

All URLs read 2026-10-12.

• Cortex Analyst, Snowflake Documentation — semantic views and why they exist, predefined join paths, the model preference order, "You cannot choose a model directly", multi-turn limitations, and the recommendation to transition to Cortex Agents.
• Overview of semantic views, Snowflake Documentation — logical tables, facts, dimensions, metrics, relationships, and the authoring paths including Autopilot, SQL DDL and YAML or BI-file import.
• Cortex Analyst evaluations, Snowflake Documentation — verified queries as ground truth, judges running on Cortex LLM inference, the v1/v2/v3 SQL correctness metric, and the temporary removal of evaluation queries from the semantic view.
• Snowflake Cortex Analyst: Evaluating Text-to-SQL Accuracy for Real-World BI — the 90%+ claim, the internal 150-question suite, the GPT-4o comparison and the 51% figure. Dated 29 August 2024.
• Genie Agents, Databricks Documentation — what a Genie Agent is, the rename from Genie Spaces, and what gets configured in one.
• Tune Genie Agent quality, Databricks Documentation — general instructions, parameterized example queries, trusted assets and verified answers, the 100-instruction and 200-snippet limits, and automatic saving of schema PK/FK as join relationships.
• Curate an effective Genie Agent, Databricks Documentation — "Aim for five or fewer tables", the 50-table limit, and prejoining into views or metric views.
• Building Confidence in Your Genie Space with Benchmarks and Ask for Review — Databricks' own benchmark feature marks an answer Correct only when the query result exactly matches the reference result.
• The Power of RLVR: Training a Leading SQL Reasoning Model on Databricks — 73.5% on BIRD's single-model, single-generation private test track, the Qwen 2.5 32B Coder Instruct base, and the caveat that BIRD does not capture the breadth of Genie. Dated 30 July 2025.
• BIRD-SQL benchmark — 12,751 question-SQL pairs, 95 databases, 33.4 GB, 37 domains, the 500-example mini-dev set, execution accuracy, and 92.96% human performance on the dev set.
• Spider 1.0, Yale LILY — 10,181 questions over 200 databases, exact set match without values and execution accuracy, and the February 2024 notice that submissions and leaderboard updates have stopped.
• Spider 2.0 — 632 enterprise-derived text-to-SQL workflow problems, the lite and snow variants, and databases running past a thousand columns.
• mnemiq benchmark results and methodology, Agentic Fabriq — the source of Table 1, the stability figures, the 27 Spider 2.0 refusals and their causes, the foreign-key replay, the Unity Catalog relationship gaps and the 30-table workaround. First-party.
• beacon, the grader and tracker — the grading rules behind both metrics, at v7 for the published numbers.
• mnemiq — the engine in the top row: a model proposes SQL, deterministic checks on shape, access, dialect and query plan decide whether it runs.
• Text-to-SQL evaluation taxonomy — the eight dimensions, including grading rule, answer shape, refusal and stability. Ours, vendor-neutral.