Mollick: expert-level work benchmarks are brutally hard, and ARC-AGI, GDPval, METR are saturated
emollick · x · 2026-11-02
Ethan Mollick praises SemiAnalysis for pulling off something valuable: benchmarks of expert-level work are incredibly hard to do, and SemiAnalysis is managing it.
His take: this is especially useful right now because many major benchmarks — ARC-AGI, GDPval, METR Long Tasks — have become completely saturated, so the field needs harder evaluations that can actually differentiate top-tier model performance.
More from Models
- Claim: Opus 5.5 recreates OpenAI's new promo entirely in code with Three.js, GLSL and an SDF heroine — FinanceYF5 · 2026-11-02
- HalachaBench sparks debate: is Talmudic Q&A testing knowledge or reasoning? — teortaxesTex · 2026-11-02
- Leak: Anthropic's unreleased Claude Mythos 5.1 surfaces; Fable 5.5 launch expected this week — haider1 · 2026-11-02
- Community prediction of Opus 5.5 launching in October called 'surprisingly accurate' — dejavucoder · 2026-11-02
- WhatsApp officially congratulates Kimi: 'you made history' — crystalsssup · 2026-11-02
- Remember when everyone called GPT-Astra AGI? Nobody uses it after Opus 5.5 — ns123abc · 2026-11-02