Mollick: public AI benchmarks are broken and underestimate AI abilities
emollick · x · 2026-09-16
Ethan Mollick argues the state of public AI benchmarking is "dire" and undermining our ability to gauge current AI. Per the cited paper, most famous benchmarks are saturated, while the non-saturated ones are riddled with errors that vastly underestimate AI abilities.
Related event: Mollick: public AI benchmarks are broken and underestimate models(2 posts)→
More from Models
- Open-source VoxLock tops Krisp voice isolation benchmark with 4.95% WER — Arbaz Siddiqui · 2026-11-02
- Anthropic's 85% inference margin sparks debate on open source's price-pressuring role — code_star · 2026-11-02
- User reports xAI appears to have quietly raised weekly usage limits, dropping his meter from 93% to 47% — XFreeze · 2026-11-02
- Meituan's LongCat-2.5 lands as 1.6T MoE agent model, with DeepSeek copycat speculation — teortaxesTex · 2026-11-02
- Asked what AI firms want you to stop owning, Claude said: your own compute — alex_verem · 2026-11-02
- Is Hermes Still Worth It? Codex's $200 Tier Now Overlaps Most of Its Value — Foreign_Lead_3582 · 2026-11-02