Ethan Mollick: Frontier AI Benchmarks Are Losing Human Baselines
emollick · x · 2026-07-31
Wharton Professor Ethan Mollick points out that as benchmarks testing frontier AI capabilities become more complex, the industry is losing one of the most important aspects of benchmarking: comparisons to humans.
He argues that validated benchmarks must include human (ideally multiple humans) baselines. Although obtaining these human baselines is increasingly difficult and expensive, it remains crucial for accurately assessing AI's true capabilities. He also lists currently unsaturated benchmarks, including ARC-AGI, GDPval, and METR long-horizon tasks.
Related event: Human Baselines Missing in AI Evaluations, Highlighting Human-AI Synergy(4 posts)→
More from AGI Musings
- Duke Study of 4,439 People Finds Coworkers Quietly Rate AI Users as Lazier — aakashgupta · 2026-10-16
- DSPy Author's $70 O'Reilly Book Rejected by Engineer Who Now Just Tells Claude What to Do — lateinteraction · 2026-10-16
- 'A machine proving new math, a car driving up a wall': AI researcher on how weird it's getting — rand_longevity · 2026-10-16
- Yacine on training: 'Neural nets are lazy stupid idiots — I build hells to make them ubermensch' — yacineMTB · 2026-10-16
- Jeff Clune: Moving the Goalposts Will Be Humanity's Last Job — jeffclune · 2026-10-16
- Replit CEO Amjad Masad Says the Future of Work Is the 'Self-Driving Company' — TechCrunch AI · 2026-10-16