Epoch AI audits 15 benchmarks: 4 verified, 9 flawed, and pushback dismissed
hugobowne · x · 2026-10-27
Epoch AI launched Benchmark Reviews, auditing 15 benchmarks to start: 4 verified, 9 flawed, and 2 with too little information to review.
Hamel Husain responds to three common objections — that benchmark owners already know and are fixing the flaws, that critics should contribute instead, and that flawed verdicts are too binary. He finds the pushback misguided: these benchmarks move millions of dollars of token spend, many people blindly trust that they generalize to their own products, and look no further than headline numbers. Whenever a new model drops, thousands of videos plot the pareto frontier using one benchmark as the primary quality metric without questioning it.
He argues the field is better served by inviting skepticism and encouraging people to form an independent view of model capability.
More from Research
- Wolfram at ARC Prize: discovery means finding reducibility pockets in irreducible computation — mhmazur · 2026-10-27
- BEACON paper accepted to NeurIPS 2026: learns per-sample trust weights for cross-domain robot policy co-training — hankyang94 · 2026-10-27
- Beyond MaxSim: Researchers Push for Richer Interactions at Scale in Embeddings — IgorCarron · 2026-10-27
- Recursive SI trained a model from scratch in 4 months using auto research — altryne · 2026-10-27
- Nestor outperforms matched-sparsity MoE baselines in pretraining and long-context finetuning — teortaxesTex · 2026-10-27
- kohya's kv-cache Control LoRA Implementation Cuts MOSS-Image Overhead to Half of Concat — bdsqlsz · 2026-10-27