Epoch AI Launches Benchmark Audits: 9 of 15 AI Benchmarks Found Flawed
Jsevillamol · x · 2026-09-20
Epoch AI announced Benchmark Reviews, a new initiative to audit AI benchmarks. Of the first 15 reviewed, 4 were Verified, 9 judged Flawed, and 2 had insufficient information to review.
The DeepSWE team responded, welcoming scrutiny: as their first benchmark, they had shared all tasks and grading results openly. They added they have since improved their methods internally and believe no existing benchmarks accurately reflect current model capabilities—upcoming work will address this.
Related event: Epoch AI's First Benchmark Audit Flags 9 of 15 as Flawed(22 posts)→
More from Research
- Harvard Professors Admit AI Article Reviews Now Beat the Best Humans — iamfakhrealam · 2026-11-04
- Pre-pretraining gains traced to embedding init: N(0,0.02) alone lifts perplexity 9% — LauraRuis · 2026-11-04
- kalomaze shares loss curve of larger-backbone run on DeepSeek V3.1 base, eval loss down to ~0.82 — kalomaze · 2026-11-04
- Hobbyist rewrites Qwen 0.6B weights with 39.8M tokens, no gradients, in 3h12m — Altruistic_Heat_9531 · 2026-11-04
- SpecPilot agent boosts speculative decoding from 4.46× to 10.87×, no retraining — pmttyji · 2026-11-04
- Autonomous agent designs target-specific MSAs to improve AlphaFold 3 complex prediction — SJTU · 2026-11-04