Moritz Hardt: AI benchmarks are fundamentally broken but fixable

asusarla · x · 2026-10-18

Quanta Magazine interviews computer scientist Moritz Hardt, author of a new book on benchmarks, who argues AI benchmarks are both the engine and the distortion of the field.

He calls benchmarks "the mistake that made machine learning": simple rulers that ended up dominating the field's behavior.
Historical evidence: DARPA's 1987 speech benchmark helped trigger a 25-year AI winter; ImageNet in 2012 kicked off the deep learning revolution.
He finds OpenAI and Anthropic's benchmark claims misleading — today's benchmarks are created and solved with clockwork regularity while claiming to measure everything from domain expertise to "general intelligence."
His stance: benchmarks aren't unfixable, but benchmarking must become a science rather than a dark art — otherwise we can't know how good models really are.

Original post →

Moritz Hardt: AI benchmarks are fundamentally broken but fixable

More from Models