Pervasive Annotation Errors Break Text-to-SQL Benchmarks and Leaderboards
Tengjun Jin, Yoojin Choi, Yuxuan Zhu, Daniel Kang
cs.AI, cs.DB
2026-01-14
An audit finds 52.8% of BIRD Mini-Dev and 62.8% of Spider 2.0-Snow annotations wrong; after correcting 100 examples, 16 leaderboard agents shift up to 9 rank positions.
Text-to-SQL practitioners pick systems by public leaderboards, mostly BIRD and Spider. These benchmarks pair human-written questions with gold SQL queries, and scoring runs both the generated and the gold SQL against a database and compares results (Execution Accuracy, EX). When the gold query is wrong, every downstream score and rank is wrong.
A prior audit by the Arcwise team put the error rate of BIRD Mini-Dev at 32.3%, with a separate report finding 49% in the financial domain. This UIUC team turns the audit into a systematic pipeline: an AI agent screens every example, SQL experts adjudicate, a random 100-example subset gets corrected, and all 16 reproducible open-source agents on the BIRD leaderboard get re-run. The measured error rate is past half, and the leaderboard largely falls apart. A meaningful share of the relative gaps reported on BIRD likely traces to the annotations rather than the methods.
The screening tool, SAR-Agent (SQL Annotation Reviewer), runs on OpenAI o3 with up to 30 iterations per example. Instead of a single-turn LLM judgment, it mimics how analysts work through exploratory data analysis and treats database execution results as evidence:
Humans work in three steps: SAR-Agent produces reports; two authors independently verify each flagged case, with all four authors resolving disagreements; the experts then extend review to similar examples based on recurring patterns. Errors fall into four classes: E1, gold SQL contradicting the question's logic (BETWEEN where the question demands strict inequality); E2, misunderstanding the data or schema, the most common class in both benchmarks at roughly 58% of erroneous examples; E3, wrong domain knowledge, such as treating K-12 as grades 1 through 12 and dropping kindergarten; E4, ambiguous questions.
Corrections use SAPAR, a pipeline that embeds SAR-Agent into the standard annotation workflow: the agent reviews, humans verify, revisions loop back, up to three rounds. Of the 100 sampled examples (62 easy, 28 moderate, 10 challenging), 48 were corrected: 41 SQL queries, 19 questions, 17 external-knowledge notes, 1 schema. Six databases also got distinguishing rows inserted so that wrong SQL can no longer produce matching results by accident.
| Benchmark | Metric | Result |
| BIRD Mini-Dev | Annotation error rate | 52.8% (263/498); prior audit reported 32.3% |
| Spider 2.0-Snow | Annotation error rate | 62.8% (76/121 examples with public gold SQL) |
| SAR-Agent | Precision | 83% on BIRD, 89% on Spider 2.0-Snow |
| SAR-Agent | Cost per example | $0.44 on BIRD (5.1 steps), $1.11 on Spider 2.0-Snow (7.6 steps) |
SAR-Agent hits 85.7% of the 161 errors Arcwise reported and finds 41.6% more errors beyond them. The Spider 2.0 team had already run two LLM-assisted validation rounds, fixing 45% and then 5% of examples, yet 62.8% of the audited subset remains wrong, worse than BIRD.
Re-evaluating the 16 agents on the 100-example subset, original vs. corrected (ranks within these 16):
| Agent | Original rank | Corrected rank | EX change |
| CHESS | 7 | tied 1st | 62% to 81%, +30.6% relative |
| MAC-SQL | 14 | 5 | +29.3% relative |
| Contextual-SQL | 1 | 7 | declined |
| OpenSearch-SQL | 2 | 11 | declined |
| GSR | 4 | 13 | +3% relative, overtaken |
| SFT CodeS-15B | 15 | 16 | 56% to 52%, -7.1% relative |
Across all 16 agents, relative EX changes span -7% to +31% (12 improve); rank shifts average 5 positions and reach 9. The correlation analysis pins the cause on annotations: rankings on the original subset track the full Dev set closely (Spearman 0.85, p = 3.26e-5), so sampling is not the issue, while original vs. corrected rankings correlate weakly (0.32, p = 0.23). The reshuffle comes from the annotation fixes, not sampling noise.
For anyone building on text-to-SQL: BIRD's relative rankings should not be taken at face value, and agents like CHESS may have been undervalued by wrong gold answers. The broader lesson is methodological. Benchmarks scored by execution comparison should be audited by execution too, and SAR-Agent plus expert adjudication costs $0.44 to $1.11 per example, a pipeline that transfers to other execution-based benchmarks. The Spider 2.0 numbers also show that pure LLM review is not enough: after two LLM-assisted correction rounds, it still carries the higher error rate of the two benchmarks.
Corrections cover only a 100-example random subset. The original subset ranks correlate strongly with the full Dev set, but whether the corrected subset represents a fully corrected Dev set is not directly verified; all 16 agents' shifts are subset numbers. Final correctness labels were adjudicated by the four authors, with domain knowledge cross-checked against sources like Wikipedia and CDC, and no external arbitration; SAR-Agent's suggested corrected queries were not separately evaluated for accuracy, which the paper itself acknowledges. Spider 2.0-Snow has 547 examples, but only the 121 with public gold SQL were audited. Corrections also included inserting rows into databases, which changes the evaluated artifact itself; an ablation shows minimal ranking impact (0.95), but strictly speaking the corrected benchmark is no longer the same database.