Over Half of BIRD's Labels Are Wrong: Fixing Them Reshuffles the Text-to-SQL Leaderboard

Pervasive Annotation Errors Break Text-to-SQL Benchmarks and Leaderboards

Tengjun Jin, Yoojin Choi, Yuxuan Zhu, Daniel Kang

cs.AI, cs.DB

2026-01-14

An audit finds 52.8% of BIRD Mini-Dev and 62.8% of Spider 2.0-Snow annotations wrong; after correcting 100 examples, 16 leaderboard agents shift up to 9 rank positions.

What problem this solves

Text-to-SQL practitioners pick systems by public leaderboards, mostly BIRD and Spider. These benchmarks pair human-written questions with gold SQL queries, and scoring runs both the generated and the gold SQL against a database and compares results (Execution Accuracy, EX). When the gold query is wrong, every downstream score and rank is wrong.

A prior audit by the Arcwise team put the error rate of BIRD Mini-Dev at 32.3%, with a separate report finding 49% in the financial domain. This UIUC team turns the audit into a systematic pipeline: an AI agent screens every example, SQL experts adjudicate, a random 100-example subset gets corrected, and all 16 reproducible open-source agents on the BIRD leaderboard get re-run. The measured error rate is past half, and the leaderboard largely falls apart. A meaningful share of the relative gaps reported on BIRD likely traces to the annotations rather than the methods.

Method

The screening tool, SAR-Agent (SQL Annotation Reviewer), runs on OpenAI o3 with up to 30 iterations per example. Instead of a single-turn LLM judgment, it mimics how analysts work through exploratory data analysis and treats database execution results as evidence:

Humans work in three steps: SAR-Agent produces reports; two authors independently verify each flagged case, with all four authors resolving disagreements; the experts then extend review to similar examples based on recurring patterns. Errors fall into four classes: E1, gold SQL contradicting the question's logic (BETWEEN where the question demands strict inequality); E2, misunderstanding the data or schema, the most common class in both benchmarks at roughly 58% of erroneous examples; E3, wrong domain knowledge, such as treating K-12 as grades 1 through 12 and dropping kindergarten; E4, ambiguous questions.

Corrections use SAPAR, a pipeline that embeds SAR-Agent into the standard annotation workflow: the agent reviews, humans verify, revisions loop back, up to three rounds. Of the 100 sampled examples (62 easy, 28 moderate, 10 challenging), 48 were corrected: 41 SQL queries, 19 questions, 17 external-knowledge notes, 1 schema. Six databases also got distinguishing rows inserted so that wrong SQL can no longer produce matching results by accident.

Results

BenchmarkMetricResult
BIRD Mini-DevAnnotation error rate52.8% (263/498); prior audit reported 32.3%
Spider 2.0-SnowAnnotation error rate62.8% (76/121 examples with public gold SQL)
SAR-AgentPrecision83% on BIRD, 89% on Spider 2.0-Snow
SAR-AgentCost per example$0.44 on BIRD (5.1 steps), $1.11 on Spider 2.0-Snow (7.6 steps)

SAR-Agent hits 85.7% of the 161 errors Arcwise reported and finds 41.6% more errors beyond them. The Spider 2.0 team had already run two LLM-assisted validation rounds, fixing 45% and then 5% of examples, yet 62.8% of the audited subset remains wrong, worse than BIRD.

Re-evaluating the 16 agents on the 100-example subset, original vs. corrected (ranks within these 16):

AgentOriginal rankCorrected rankEX change
CHESS7tied 1st62% to 81%, +30.6% relative
MAC-SQL145+29.3% relative
Contextual-SQL17declined
OpenSearch-SQL211declined
GSR413+3% relative, overtaken
SFT CodeS-15B151656% to 52%, -7.1% relative

Across all 16 agents, relative EX changes span -7% to +31% (12 improve); rank shifts average 5 positions and reach 9. The correlation analysis pins the cause on annotations: rankings on the original subset track the full Dev set closely (Spearman 0.85, p = 3.26e-5), so sampling is not the issue, while original vs. corrected rankings correlate weakly (0.32, p = 0.23). The reshuffle comes from the annotation fixes, not sampling noise.

Why it matters

For anyone building on text-to-SQL: BIRD's relative rankings should not be taken at face value, and agents like CHESS may have been undervalued by wrong gold answers. The broader lesson is methodological. Benchmarks scored by execution comparison should be audited by execution too, and SAR-Agent plus expert adjudication costs $0.44 to $1.11 per example, a pipeline that transfers to other execution-based benchmarks. The Spider 2.0 numbers also show that pure LLM review is not enough: after two LLM-assisted correction rounds, it still carries the higher error rate of the two benchmarks.

Limitations

Corrections cover only a 100-example random subset. The original subset ranks correlate strongly with the full Dev set, but whether the corrected subset represents a fully corrected Dev set is not directly verified; all 16 agents' shifts are subset numbers. Final correctness labels were adjudicated by the four authors, with domain knowledge cross-checked against sources like Wikipedia and CDC, and no external arbitration; SAR-Agent's suggested corrected queries were not separately evaluated for accuracy, which the paper itself acknowledges. Spider 2.0-Snow has 547 examples, but only the 121 with public gold SQL were audited. Corrections also included inserting rows into databases, which changes the evaluated artifact itself; an ablation shows minimal ranking impact (0.95), but strictly speaking the corrected benchmark is no longer the same database.

Terms

Source

What people are saying

Related papers

All paper explainers