LuthorSQL: A One-Pass Text-to-SQL Model as System 1 for a Slower LLM Agent
Ibrahim Memon, Armin Karamzade, M. Hanif Jhaveri, Roy Fox
cs.CL, cs.AI, cs.DB, cs.LG
2026-11-13
LuthorSQL fills a typed SQL plan in one forward pass on a 7B backbone, hits 68.03% on BIRD's hidden test, and cuts Claude Opus API cost 33% in a cascade at equal accuracy.
Tool-using LLM agents are the most accurate text-to-SQL systems around: run with a read-only SQL tool, Claude Opus 5.5 answered 77.26% of 796 held-out BIRD questions. The bill is the problem. At list price that is roughly two cents per question, over $70,000 a year at 10,000 questions a day, with every query shipped to an external API.
The motivating observation is plain: of 311 questions Opus answered correctly in a 400-question sample, the authors' one-pass 7B model also got 228 right. Most organizational questions are routine. So they borrow Kahneman's framing: a fast local System 1 handles the routine, and a slow System 2 agent gets whatever System 1 is unsure about.
LuthorSQL abandons token-by-token generation and returns to slot filling, rebuilt on a modern backbone:
Both adapters share one set of frozen backbone weights, loaded once. This is one 7B model with two small adapters, not an ensemble.
| Setting | Metric | Result |
| LuthorSQL alone, BIRD hidden test (1,789 q) | Execution accuracy | 68.03%, 0.84 s per question on one H100 |
| Same model, 8-bit quantized | Accuracy / footprint | Matches GPU accuracy in 9 GB (159 vs 158 correct on 239 questions, on an M3 laptop) |
| Zero-shot dialect transfer, BIRD dev | Execution accuracy | 57.1–59.3% on Snowflake, BigQuery, MySQL, PostgreSQL, DuckDB vs 60.17% in SQLite |
| Cascade vs direct Opus API, 227 post-cutoff questions | Cost / accuracy | 33.3% cheaper, 95.15% vs 95.59% (not significant) |
| Cascade, RESERVED (348 q, three runs) | Cost | 24.8% / 32.9% / 32.5% saved, accuracy within 1.2 points of Opus |
| Contamination probe: Opus, no schema | Gold SQL reproduced verbatim | 48–92% |
About half of all questions never leave the machine: 49.1% handled locally on RESERVED, 48.1% on the external set (KaggleDBQA + Spider-Realistic, 693 questions). The k-best machinery pays for itself: a single greedy plan takes 0.13 s and scores 49.12% on DECISION, and expanding to 32 candidates adds 0.7 s for 9.93 points, or $0.64 per 1,000 queries.
The contamination half deserves its own read. Given only a question and a database name, with no schema and no tool, Opus reproduces 48–92% of gold SQL verbatim across four public splits, down to dataset-specific alias conventions and backtick formatting. Thin-prompted LLM scores on these benchmarks partly measure recall. The agent's skill survives the controls, though: with every table and column renamed and the questions paraphrased, accuracy is unchanged (72.88% vs 72.32%, p = 1.0), and on eight databases published after its training cutoff, tool-assisted accuracy reaches 95.6–97.4%.
For teams building natural-language database interfaces, three things here are directly usable: half the queries stay on premises, API cost drops 16–34% with no significant accuracy loss, and System 1 fits in 9 GB, so hardware the organization already owns can host it. One checkpoint serves six SQL engines through zero-shot translation. The design also names where small models fail (syntax errors, hallucinated literals, join logic) and closes each gap structurally rather than with more sampling.
The memorization check deserves to become standard practice; the authors recommend that every LLM text-to-SQL baseline run it.
Be clear about what this is not: 68.03% standalone is nine points behind the frontier agent. The cascade saves money; it does not beat the agent. This is a solid engineering trade, not an accuracy leap.
The authors' own list is long. The plan grammar covers 84.6% of BIRD dev gold queries; anything needing CTEs, window functions, or correlated subqueries goes to the agent. The cascade has not been run on BIRD's hidden test set, and the held-out tests are small (8, 27, and 8 databases), so intervals are wide; the first pre-registered RESERVED run missed its own cost bar by 0.1 point (29.9% against a 30% target). Costs are list prices applied to usage logs, not invoices, and part of the saving comes from running fewer Claude Code sessions.
Three things look weaker on a close read. Out-of-grammar questions in the DECISION experiments were identified using gold SQL, information a deployed system would not have, so those cascade results assume a perfect detector. VALHARD's hard-negative categories were designed after inspecting errors on the same split used to evaluate it, so its +1.76-point gain may be optimistic, and it missed the pre-written significance bar (p = 0.19). BIRD's hidden test score sits 8.64 points above the dev score and the authors say they cannot explain the gap. The "held-out" label applies to the adapters only, since the OmniSQL backbone was fine-tuned on all of BIRD train. English only, foreign-key joins only, and the agent itself was tested on SQLite only.
The ethics statement discloses that Claude wrote much of the experiment code and helped prepare the manuscript, with the first author checking every number against logs. Transparent, and worth keeping in mind when weighing the results.