Audit finds 9 of 15 AI benchmarks flawed: 45.5% of Terminal Bench 4.0 tasks broken
ajratner · x · 2026-09-18
Researchers audited 15 AI benchmarks and labeled 9 as flawed:
- Terminal Bench 4.0: 45.5% of tasks found broken after reviewing GitHub issues
- HLE: 46% of 48 randomly sampled questions were broken
- DeepSWE 1.1: a bug was found that can break grading for every task
Responding to pushback, alexgshaw noted they continuously maintain their benchmarks and are the ones filing the GitHub issues. The audit adds to growing concerns about benchmark quality and the trustworthiness of leaderboard scores.
Related event: Epoch AI's First Benchmark Audit Flags 9 of 15 as Flawed(22 posts)→
More from Research
- Harvard Professors Admit AI Article Reviews Now Beat the Best Humans — iamfakhrealam · 2026-11-04
- Pre-pretraining gains traced to embedding init: N(0,0.02) alone lifts perplexity 9% — LauraRuis · 2026-11-04
- kalomaze shares loss curve of larger-backbone run on DeepSeek V3.1 base, eval loss down to ~0.82 — kalomaze · 2026-11-04
- Hobbyist rewrites Qwen 0.6B weights with 39.8M tokens, no gradients, in 3h12m — Altruistic_Heat_9531 · 2026-11-04
- SpecPilot agent boosts speculative decoding from 4.46× to 10.87×, no retraining — pmttyji · 2026-11-04
- Autonomous agent designs target-specific MSAs to improve AlphaFold 3 complex prediction — SJTU · 2026-11-04