New paper: error-ridden unsaturated benchmarks vastly underestimate AI capabilities
emollick · x · 2026-09-16
Ethan Mollick argues the state of public AI benchmarking is dire and undermining our ability to judge how good AI actually is. Citing a new paper, he notes most famous measures are maxed out (saturated), while the non-saturated benchmarks are riddled with so many errors that they vastly underestimate AI abilities.
Related event: Mollick: public AI benchmarks are broken and underestimate models(2 posts)→
More from Research
- GDPx: SemiAnalysis's frontier benchmark across the 11 most valuable domains — rohanpaul_ai · 2026-11-02
- SemiAnalysis launches GDPx: best frontier model scores just 12% on expert work — rohanpaul_ai · 2026-11-02
- Detecting covert instruction-following via internal representations parallels J-space work — dioscuri · 2026-11-02
- Xiaomi MiMo's CodeSpecWeave trains coding agents on interdependent specs for long-horizon coding — XiaomiMiMo · 2026-11-02
- Mistral Large 4 writes a quantum chemistry SCF solver from scratch with one short prompt — MistralAI · 2026-11-02
- Hunting for the most concrete "curing cancer with AI" roadmap of 2025 — iskander · 2026-11-02