Epoch AI audits AI benchmarks: 9 of 15 reviewed benchmarks found flawed
joecole · x · 2026-09-18
Epoch AI launched Benchmark Reviews, a new initiative to audit AI benchmarks, starting with 15 benchmarks: only 4 are Verified, 9 are Flawed, and 2 lack sufficient information for review. The finding raises serious questions about the reliability of widely cited benchmark scores.
Related event: Epoch AI's First Benchmark Audit Flags 9 of 15 as Flawed(22 posts)→
More from Research
- PixelBench: testing multimodal models pixel-by-pixel instead of with QA — bdsqlsz · 2026-11-04
- François Chollet: modeling brain wiring as an engineering problem constrained by genome size and time — CatAstro_Piyush · 2026-11-04
- OffsecCyber dataset combines HackerOne reports and CVEs to train pentesting LLMs — AccBalanced · 2026-11-04
- Monofact rate in training data governs LLM scientific discovery, preprint finds — CatAstro_Piyush · 2026-11-04
- qwenTPU compiles a TPU into Qwen3.5 weights, achieving exact integer matmul — CatAstro_Piyush · 2026-11-04
- Jev model improves ICL demonstration retrieval via distillation into a lightweight MLP — kalyan_kpl · 2026-11-04