A VLDB 2026 paper finds over 50% annotation errors in BIRD and Spider 2.0-Snow
ddkang · x · 2026-07-28
- A VLDB 2026 paper on text-to-SQL agents reports more than 50% annotation error rates in BIRD and Spider 2.0-Snow.
- The authors say those errors can change measured performance by up to 19%.
- The result is framed as a warning that benchmark and leaderboard numbers may be far less reliable than they appear.
More from Research
- Minerva paper details: ncRNA arrays template short DNA hairpins — BrianHie · 2026-09-23
- Genome language models uncover new class of reverse-transcriptase mechanisms — BrianHie · 2026-09-23
- Mathematician shares a cheap 4-step heuristic for hyperparameter tuning — dejanseo · 2026-09-23
- Burkov skew AI hype: 'deterministic LLMs' and 'first agents' are old tricks rebranded — burkov · 2026-09-23
- Continuous diffusion beats discrete on random k-SAT, proposed as standard benchmark — ArashVahdat · 2026-09-23
- Grady Booch: Contemporary AI Still Lacks Abductive Reasoning, Just 'Next-Token Prediction' — Grady_Booch · 2026-09-23