Audit flags 45.5% of Terminal Bench 4.0 tasks as broken; maintainers push back on rigor
DimitrisPapail · x · 2026-09-19
An audit of 15 benchmarks labeled 9 as flawed: 45.5% of Terminal Bench 4.0 tasks were judged broken after reviewing GitHub issues, 46% of 48 randomly sampled HLE questions were broken, and a bug in DeepSWE 1.1 could break grading for every task.
Terminal Bench's Ryan Marten pushed back, arguing the flaws affect <3% of leaderboard rollouts and wouldn't move scores beyond reported confidence intervals. He noted the "flaws" were pulled from already-triaged public GitHub issues — no new issues were filed — and that TB publishes raw receipts (per-trial logs, trajectories, rewards) for every run. His point: all software tasks have bugs; what matters is impact magnitude and the mechanism for catching them.
Related event: Epoch AI's First Benchmark Audit Flags 9 of 15 as Flawed(22 posts)→
More from Research
- PixelBench: testing multimodal models pixel-by-pixel instead of with QA — bdsqlsz · 2026-11-04
- François Chollet: modeling brain wiring as an engineering problem constrained by genome size and time — CatAstro_Piyush · 2026-11-04
- OffsecCyber dataset combines HackerOne reports and CVEs to train pentesting LLMs — AccBalanced · 2026-11-04
- Monofact rate in training data governs LLM scientific discovery, preprint finds — CatAstro_Piyush · 2026-11-04
- qwenTPU compiles a TPU into Qwen3.5 weights, achieving exact integer matmul — CatAstro_Piyush · 2026-11-04
- Jev model improves ICL demonstration retrieval via distillation into a lightweight MLP — kalyan_kpl · 2026-11-04