VLDB 2026 Paper Audits BIRD and Spider 2.0-Snow, Finding >50% Annotation Errors

A VLDB 2026 paper systematically audits mainstream text-to-SQL benchmarks, finding annotation error rates of 52.8% in BIRD Mini-Dev and 62.8% in Spider 2.0-Snow. The study shows that benchmark data annotation quality is critical for leaderboard reliability; current LLM scores and rankings are severely distorted. The authors urge developers to verify data quality rather than blindly trust rankings.

Confirmed

The research team uses a three-stage audit process: first, an automatic diagnostic system SAR-Agent generates reports for each sample; then human SQL experts adjudicate flagged cases; finally, samples with recurring error patterns are reviewed again.

The most common error type is "insufficient data understanding," accounting for 57.79% in BIRD Mini-Dev and 57.89% in Spider 2.0-Snow.

After re-evaluating 16 open-source text-to-SQL agents on a corrected BIRD Dev subset, rankings shift by -9 to +9. For example, the former SOTA model Contextual-SQL drops from 1st place.

Why it matters

Current LLM evaluation in text-to-SQL heavily relies on these mainstream benchmarks and leaderboards. Over 50% annotation errors mean that academia and industry must prioritize dataset quality review over rankings when assessing agent performance; otherwise, misleading conclusions will be drawn.

2026-07-28 ~ 2026-07-28 · 8 related posts

Primary sources

A VLDB 2026 paper finds over 50% annotation errors in BIRD and Spider 2.0-Snow — ddkang · 2026-07-28 02:14 (UTC+08:00)
Two widely used text-to-SQL benchmarks are reported to have 52.8% and 62.8% error rates — ddkang · 2026-07-28 02:14 (UTC+08:00)
Corrected text-to-SQL data shifts 16 open-source agent rankings by up to nine places — ddkang · 2026-07-28 02:14 (UTC+08:00)

Study finds BIRD and Spider 2.0-Snow text-to-SQL benchmarks exceed 50% annotation error — ddkang · 2026-07-28
Text-to-SQL benchmark audit uses SAR-Agent plus human SQL expert review — ddkang · 2026-07-28
[source] A VLDB 2026 paper finds over 50% annotation errors in BIRD and Spider 2.0-Snow — ddkang · 2026-07-28
[source] Two widely used text-to-SQL benchmarks are reported to have 52.8% and 62.8% error rates — ddkang · 2026-07-28
[source] Corrected text-to-SQL data shifts 16 open-source agent rankings by up to nine places — ddkang · 2026-07-28
Text-to-SQL benchmark audit finds data-understanding errors as the most common pattern — ddkang · 2026-07-28
Benchmark developers are told to verify text-to-SQL data quality before trusting rankings — ddkang · 2026-07-28
A text-to-SQL paper says benchmark annotation errors can distort rankings and scores — ddkang · 2026-07-28