Epoch launches Benchmark Reviews, auditing 15 AI benchmarks: only 4 verified, 9 flawed
stochasticchasm · x · 2026-09-18
- Epoch AI launched Benchmark Reviews, a new initiative to audit AI benchmarks, starting with 15 benchmarks: 4 Verified, 9 Flawed, and 2 with insufficient information for review.
- The finding that most popular benchmarks are flawed is a big deal for anyone doing agent evals.
- Community reaction: Arthur Bresnu called it "the state of agentic evals in one tweet" and hoped it would force people to take benchmark quality more seriously.
Related event: Epoch AI's First Benchmark Audit Flags 9 of 15 as Flawed(22 posts)→
More from Research
- Harvard Professors Admit AI Article Reviews Now Beat the Best Humans — iamfakhrealam · 2026-11-04
- Pre-pretraining gains traced to embedding init: N(0,0.02) alone lifts perplexity 9% — LauraRuis · 2026-11-04
- kalomaze shares loss curve of larger-backbone run on DeepSeek V3.1 base, eval loss down to ~0.82 — kalomaze · 2026-11-04
- Hobbyist rewrites Qwen 0.6B weights with 39.8M tokens, no gradients, in 3h12m — Altruistic_Heat_9531 · 2026-11-04
- SpecPilot agent boosts speculative decoding from 4.46× to 10.87×, no retraining — pmttyji · 2026-11-04
- Autonomous agent designs target-specific MSAs to improve AlphaFold 3 complex prediction — SJTU · 2026-11-04