VLDB Paper Reveals Over 50% Mislabeled Rates in Major Text-to-SQL Benchmarks

A VLDB 2026 paper systematically audited mainstream text-to-SQL benchmarks, revealing mislabeling rates of 52.8% for BIRD Mini-Dev and 62.8% for Spider 2.0-Snow. The study highlights that annotation quality is vital for leaderboard reliability, urging developers to audit datasets rather than blindly trust rankings.

Confirmed

The research team employed a three-stage audit process: first using the SAR-Agent automated diagnostic system to generate reports for each sample, followed by manual adjudication of flagged cases by human SQL experts, and finally a look-back review of samples with repetitive error patterns. Regarding error distribution, mislabels caused by "insufficient data understanding" were the most common, accounting for 57.79% and 57.89% of the errors in BIRD Mini-Dev and Spider 2.0-Snow, respectively.

Why It Matters

Current evaluations of LLMs on text-to-SQL tasks rely heavily on these mainstream benchmarks and leaderboards. With over half of the labels being incorrect, existing model performances and rankings may be severely distorted. This compels both academia and industry to prioritize dataset quality audits over raw rankings when evaluating agent capabilities.

2026-07-28 ~ 2026-07-28 · 7 related posts

Primary sources

Two widely used text-to-SQL benchmarks are reported to have 52.8% and 62.8% error rates — ddkang · 2026-07-28 02:14 (UTC+08:00)
Text-to-SQL benchmark audit uses SAR-Agent plus human SQL expert review — ddkang · 2026-07-28 02:14 (UTC+08:00)
Text-to-SQL benchmark audit finds data-understanding errors as the most common pattern — ddkang · 2026-07-28 02:14 (UTC+08:00)

Study finds BIRD and Spider 2.0-Snow text-to-SQL benchmarks exceed 50% annotation error — ddkang · 2026-07-28
[source] Text-to-SQL benchmark audit uses SAR-Agent plus human SQL expert review — ddkang · 2026-07-28
A VLDB 2026 paper finds over 50% annotation errors in BIRD and Spider 2.0-Snow — ddkang · 2026-07-28
[source] Two widely used text-to-SQL benchmarks are reported to have 52.8% and 62.8% error rates — ddkang · 2026-07-28
[source] Text-to-SQL benchmark audit finds data-understanding errors as the most common pattern — ddkang · 2026-07-28
Benchmark developers are told to verify text-to-SQL data quality before trusting rankings — ddkang · 2026-07-28
A text-to-SQL paper says benchmark annotation errors can distort rankings and scores — ddkang · 2026-07-28