Hamel Husain flags poor benchmark quality: Epoch audits find Terminal-Bench, OSWorld 2.1 all flawed
HamelHusain · x · 2026-10-26
AI engineer Hamel Husain says the industry's reliance on public benchmarks is wildly out of step with their quality, pointing to Epoch AI's independent benchmark audit site. Findings he highlights:
- Terminal-Bench 4.0.0: 30 of 66 tasks (45.5%) had documented scoring defects; pending fixes didn't address reliability of the reviewed version.
- DeepSWE v1.1: at least 23 of 113 tasks (20.3%) had false negatives; in 78% of those cases, editing or adding tests could break the grader, with agents given no instructions to avoid such edits.
- OSWorld 2.1: 36 of 50 audited tasks (72%) contained errors — 24 false negatives, 12 false positives, 4 impossible tasks/checkpoints — potentially affecting up to 50.4% of the max score.
- HealthBench Professional: 25 of 50 sampled tasks (50%) had clear errors, including 17 false negatives, 8 false positives, and 6 tasks failing to measure the claimed capability.
His advice: actually run benchmarks and inspect grader failures before trusting them. He plans to publish a fuller write-up as he gathers more data.
More from Models
- Iranian users report Qwen chat access blocked; blogger says sanctions aren't Alibaba's call — dotey · 2026-10-26
- Hugging Face releases Carbon-X, an open 14.8B DNA language model that reads genomes in chat — _lewtun · 2026-10-26
- Users annoyed as ChatGPT keeps lecturing them with "Kevin, ..." openers — kevinkern · 2026-10-26
- OCRia-1: 4B doc-parsing VLM hits 0.87 mean ParseBench, decodes 2-4x faster — NielsRogge · 2026-10-26
- Best sub-40B model on Terminal-Bench 4.0: Nanbeige4.3-33B at 23.8% accuracy — NielsRogge · 2026-10-26
- Frontier models are better judged by five minutes of weird jokes than 40 benchmark numbers — korymath · 2026-10-26