Human-Level Text-to-SQL via Reinforcement Learning on Verified Data, Without Pipeline Engineering
Yuxuan Zhu, Tengjun Jin, Yoojin Choi, Daniel Kang
cs.DB, cs.CL
2026-03-20
RLVR on 2.5k cleaned BIRD items plus evidence-aware rewards takes Kimi-K2.6 to 91.4–93.8% greedy and 93.0–94.2% self-consistency, matching the 92.96% human proxy.
Top BIRD systems keep stacking schema linking, dynamic prompts, iterative rewrite, and candidate selection around an LLM. The pipelines get heavier. The gap to human experts stays above 10 points. This University of Illinois group argues the ceiling is the training signal, not another inference module.
Public Text-to-SQL sets are dirty. Prior audits put errors in more than half of BIRD and Spider2 public instances. Run RLVR (reinforcement learning with verifiable rewards) on those gold queries and the model copies the bugs. The paper samples 2.5k BIRD Train items, repairs them with multi-round SQL experts, then reshapes the reward for two task-specific failure modes. A single model, with no schema linker and no reward-model selector, reaches the human proxy.
BIRD-Platinum is the data work. Five stages: an o3 agent with a SQL tool flags errors (30 executions per item), a human expert corrects from a checklist before reading the LLM report, a second expert verifies, disagreements go to a third, then two automatic gates (non-empty result, and the new SQL must differ from the old one). The design target is soundness, not completeness. The LLM flagger is precise (90.6%) and nearly blind (24.5% recall). 16.2% of first-round fixes fail the next review. 61.1% of items needed at least one edit: 52.1% gold SQL, 26.2% questions, 18.2% evidence. 1.5% were dropped as unanswerable.
Corrected gold queries get structurally heavier: subqueries rise from 0.088 to 0.26 per query, CTEs from 0 to 0.11. One error type is especially contagious. A model trained on uncorrected data used DISTINCT correctly on 5 of 30 problems that need it.
Standard RLVR then fails in two places. Matching execution results on the example database is not semantic equivalence. VeriEQL, a bounded-input SQL equivalence checker, marks 32.8% of positive rewards as false positives on Kimi-K2.6 (range 28.8–38.4%), and that rate does not fall during training. Outcome-only rewards also ignore whether the model used BIRD evidence; 8 of 33 inspected failures (24.2%) match a reading the evidence was meant to rule out. With process rewards, that share falls to 1 of 23 (4.3%).
ReViSQL-BIRD downweights a result match that VeriEQL refutes from 1 to 1-β (β=0.2), and leaves timeouts or unsupported operators (window functions, for example) unpenalized. It also requires the trace to translate each evidence line into a constraint and to audit it before the final SQL, deducting λ=0.1 per violation. Training uses the CISPO loss, LoRA rank 32, group size 16, at most five intermediate SQL turns, 3,072 tokens per turn, with an 85:15 train/validation split.
Eval is on Arcwise-Plat, 498 expert-fixed BIRD Mini-Dev items. Full repairs questions, evidence, and gold SQL. SQL repairs only gold and keeps noisy questions.
| Setup | Arcwise-Plat-SQL | Arcwise-Plat-Full |
| ReViSQL-BIRD-K2.6 greedy | 91.4% | 93.8% |
| Same, 16-way self-consistency | 93.0% | 94.2% |
| Human proxy (BIRD Test) | 92.96% | 92.96% |
| OpenSearch (GPT-5.2) | 82.9% | 88.2% |
The 95% CI for greedy Full is [91.3, 95.6]; for 16-way SQL it is [90.4, 94.9]. The 92.96% proxy sits inside both. OpenSearch costs $0.056 per query; K2.6 greedy is 37% cheaper. Four candidates already beat every single-model baseline given 32.
Clean data dominates. BIRD Train has 3.8× more items than Platinum, yet training K2.6 on Train drops the base model by 7.0 and 5.0 points on SQL and Full. Platinum raises it by 7.2 and 5.0. Reward shaping adds 2.8 and 3.8, removing 24.6% and 38.0% of the leftover errors. The same ReViSQL recipe without shaping, on Qwen3-235B-A22B, beats Train-trained RLVR by 11.0–16.1% across four benchmarks. On Spider2-SQLite, greedy hits 37.04% at $0.0077, 2.2 points above GenaSQL at about one-eighth the cost. On Snowflake-dialect Spider2-Snow, 128-way self-consistency reaches 55.58%.
The practical claim is blunt: fix the labels before you add another inference stage. RLVR on dirty gold teaches the model to omit DISTINCT and similar bugs. A single model that can issue up to five exploratory SQL calls already sits on the human proxy; schema linkers and selection models are not required to get there.
The Qwen3-235B transfer onto Spider2, including a held-out dialect, is the evidence that this is not a BIRD-only score chase.
"Human-level" is a 92.96% proxy from original BIRD Test, not a fresh human pass on Arcwise-Plat. Numbers are on a 498-item Mini-Dev variant; the hidden official test is unreported. Of 64 residual self-consistency failures, 69% are question ambiguity and 31% are model defects, 60% of those defects being a missing schema-induced filter such as NULL. Process rewards check format (did the trace emit a constraint block), not that evidence was used correctly. VeriEQL times out or refuses some operators, so false-positive rewards can only be downweighted. The recipe is demonstrated on Kimi-K2.6 and Qwen3-235B; there is no small-model result. Generation is still allowed five tool SQL turns, so "no pipeline" means no external schema linker or selector, not a single forward pass.