Why Better Schema Modeling Beats Better Text-to-SQL Models

0:00This audio is presented by Hacker Noon, where anyone can learn anything about any technology. Why better schema modeling beats better text to sequel models, by Jason L. TLDR. DBT Labs ran the same 11 questions for different ways and found that modeling the schema. With no semantic layer involved at all, moved text to sequel accuracy from 64. 5% to 90, 0%, adding a semantic layer on top of that modeled schema moved it a further 8 points. Snowflake, using a semantic model across four bird databases, measured a 21-point average lift. MotherDug pointed three frontier models at small, clean schemas with nothing but the DDL and reported 95%. Though those same runs score 58-64% under bird's strict scoring. Read side by side, these look contradictory. They aren't. None of them IS measuring the model. They're measuring how much translation works somebody already did to the schema and how forgivingly the answers were graded.

1:00Ask an AI assistant to write a query against your warehouse and it will, almost every time, produce something that runs. That's the trap. A query that executes sand returns a number looks exactly like a query that answers the question. Write up until someone downstream builds a forecast on it. Two queries. One question. Two answers. Here's the shape this takes on an ordinary warehouse, away from the benchmark leaderboards. Assass company once last month's revenue. An AI assistant, ask two reasonable questions, produces two queries. Both queries ran without error. Both came from reasonable, good faith prompts. But query be sums. The invoice total. Once for every line item attached to that invoice. A customer with a base plan, an extra seat and a usage overage contributes their entire invoice total three times, once under each product. Platform, base, doesn't show platform revenue. It shows the full value of every invoice that happened to contain a platform line. The fix is one column. Some, the line's own amount, not the invoices.

2:04Nobody wrote a bug. The join changed what one row means and kept adding as if incident. If query bees breakdown reaches a board deck before anyone reconciles it against query A, that takes an awkward meeting to walk back. This is not a model failure. Point a frontier model at this schema and it will make the same mistake confidently and articulately, because nothing in the schema says that has a different grain than, or that is only additive at the invoice level. That knowledge lives in someone's head, or it lives in a semantic layer. If it lives in neither, which query you get as a coin flip. The spread that looks like disagreement. Three of the most cited text to SQL results around, read together, look like they're arguing. Read separately, each is internally consistent, which is the tell that they're measuring different things. DBT Labs April 2026 benchmark tested four configurations over an insurance data set, 11 questions, 20 runs per model. The one people quote is text to SQL versus semantic layer on a asterisk modeled project. Claude Sonnet 4. 6 at 90, 0% against 98, 2% GPT 5, 3 codecs at 84,

3:141% against 100, 0%, the more interesting comparison is the one underneath it. Against the original normalized tables, plain text to SQL managed 64. 5% across all 11 questions. The same method against a modeled project hit 90. 0% that 25, 5 point improvement came from modeling alone, with no semantic layer anywhere in the picture, and DBT built it by prompting an LLM to write as few DBT models as possible, which produced just three. Their summary, asterisk, adding even minimal modeling on top of raw tables improved results across the board. Asterisk, snowflake semantic model evaluation, March 2025, using Claude 3. 5 Sonnet is often cited as 57% to 78% on bird. It's really an average across four bird databases, and the spread inside it matters. Went 52% right pointing arrow 83%, 63% right pointing arrow 80%, 45% right pointing arrow 70%, 69% right pointing arrow 79%.

4:22Average, and approximately 20% increase in accuracy, 21 points, from adding a semantic model, not from a newer model. MotherDuck ran the opposite experiment, pointing Claude Opus 4, 5, GPT 5, 2 and Gemini 3 Flash at 500 bird questions with, in their words, no semantic layer. No query history, no special context, just the schema. The reported result is 95%. That figure sits at the top of a four-tier evaluation ladder, and the bottom rung is a lot less flattering, 64%. 0%, train, and 58%. 2%, test, under strict bird execution matching, rising through correction of known benchmark errors and tolerance for formatting differences to 94%. 9%, 94%, 4% once in LLM judges whether the answer is defensible. Their argument is two-part. That bird's strict scoring is itself misleading, because, to score above 62% under strict rules, you have to start reproducing the

5:24benchmark's mistakes, and that, good data modeling is the semantic layer. Quote dot, so there are two variables moving here, not one, and honest reading requires separating them. Scoring strictness explains most of the distance between MotherDuck's 95% and everyone else's numbers. You cannot line that figure up against DVDs or snowflakes and treat the gap as real capability. Shema modeling explains the rest, and it's the variable you control. DBT moved text to SQL a long way with three models and no semantic layer. Snowflake moved at 21 points with a semantic model over databases they didn't restructure. MotherDuck got high marks with no layer at all, over bird databases that, as the note, average 7 tables. None of these is a claim about which model writes better SQL. There's a ceiling worth keeping in view, too. Bird's own paper reports that it's human baseline, data engineers and database students. Reach the human result of 92. 96% execution accuracy,

6:25people who write SQL for a living, working on databases of 7 tables, get roughly one question in 14 wrong. Thibar was never flawless. It's, at least as reliable as the analyst who used two on this. What the schema needs modeling actually means. Modeling sounds like an abstraction exercise. In practice, on a schema like the one above, it's answering three concrete questions before anything touches a model. What is the grain of each table, in one sentence? Is one rope or invoice? Is one rope or line on an invoice? Join them and your result sets grain becomes one rope or line item, so any of an invoice level column is now double, triple or encoding, depending on how many lines the average invoice carries. That single fact is what would have caught query B which columns carry business meaning that isn't in their name. Looks unambiguous. Does it need to exclude refunds? Is a partially refunded invoice still paid for revenue purposes? These are policy decisions someone made once, verbally, in a meeting nobody minited,

7:26and no assistant can recover them from the schema alone. Is there more than one path to the same number? Revenue might come from, from summing, or from a table of finance job populates nightly. Three legitimate paths, three numbers that won't quite agree, and none of them, wrong, they answer slightly different questions that all get asked is, what's our revenue? Just model your warehouse, is easy advice to give, the obvious objection to all of this. DVD's benchmark fixed at schema with three new models over an 11 question insurance data set. You have 400 tables, three teams who each define revenue differently, and a migration budget of zero. Model the schema properly, is not a thing you can do this quarter, and anyone who has tried knows that the modeling is the easy half, getting three departments to agree which definition wins is the hard one. That constraint is exactly why semantic layers exist. They're the retrofit path. A place to encode grain, join paths and metric definitions without restructuring the warehouse underneath.

8:27Snowflakes 21 point average came this way, over databases nobody rebuilt. Treating, clean DDL, and semantic layer, as interchangeable is true for benchmarks and false for anyone with legacy tables and a roadmap. There's also a second argument for the retrofit that the accuracy columns actively hide. In DVD's unmodeled configuration, the semantic layer scored zero, zero percent on the subset of questions requiring too many joins to resolve, not because IT answered them wrongly, but because it declined to answer at all. As DBT puts it, the semantic layer tells you it can't answer. It never returns invalid data. Text to SQL will cheerfully give you a wrong number. Plane text to SQL scored 70 to 100% on that same subset, which looks like a win until you remember that nothing in those results distinguishes a correct answer from a confident one. At coverage gap, you can see beats a coverage gap that quietly returns 3,357,000. That asymmetry appears in

9:27no accuracy column anywhere, and for a number headed to a board deck, it may matter more than the points do. What both routes share is scope. Neither DBT's three models, nor a semantic layer requires modeling the whole warehouse. Only the slice that answers the questions people actually ask. That's the practical move. Take the handful of questions your team asks weekly, model just the tables those touch, and leave the other 380 alone until someone asks them something. The benchmarks didn't find a model problem. They found a translation work problem, and measured what happens when someone does the translation once instead of leaving it to a fresh guess on every query. The checklist that transfers to your warehouse. None of the studies above tells you which side of the gap your own schema sits in. Run this instead. One. Take your five most asked business questions and write down, for each table involved, it's grain in one sentence. If you can't do that quickly, an assistant generating sequel against those tables can't either. Two. Find every column whose name promises more

10:28precision than its values deliver, especially. If needs a footnote to interpret correctly, that footnote has to live somewhere the assistant can read. A view, a semantic layer, a column comment. Not a slack thread from 2024. Three. Ask the same business question two ways and see whether you get two numbers. That's what happened above. If it happens on your warehouse, you have a modeling gap rather than an AI competence gap, and no model upgrade closes it. The numbers make the same point from three directions. 90% accuracy sounds like you solve problem until you notice the missing 10% is where double counted joins and silently redefined metrics live, invisible precisely because the query still runs. Building the judgment to catch that. Reading a join and knowing what IDD to the grain, reconciling a number against a second path before trusting it, is what Sofiarck's sequel for data analysis course is built around, with a dedicated lesson on joins and double counting early on in a closing module on auditing sequel that an AI assistant wrote. References. Gens,

11:30J, and Parago. B. 2026. April 7. Semantic layer versus text to sequel. 2026 benchmark update DBT developer blog. Access the 16th of September 2026. Zayets, T. 2025. March 31. Agentech semantic model improvement. Elevating text to sequel performance. Snowflake engineering blog. Access the 16th of September 2026. MotherDuck, your data model is the semantic layer. MotherDuck blog, undated. Access the 16th of September 2026. Lee, J. Wei, B. Koo, G. et al. 2023. Can LLM already serve as a database interface? A big bench for large scale database grounded text to SQLs. Archive. 2305.03111. Source of the 92. 96% human performance baseline on bird sequel. Access the 16th of September 2026. Sophie Arge,

12:33sequel for data analysis. Course covering filtering, aggregation, joins, CTEs, window functions, and cohort analysis. Closing with a module on auditing AI generated sequel for the silent failure modes described here. Double counting joins, null traps and misplaced filters. Thank you for listening to this Hackernoon story, read by artificial intelligence. Visit Hackernoon.com to read, write, learn and publish.