AI autonomy benchmarks may be wrong, says Ramez; Mollick adds caveats

eldonredwards · x · 2026-11-21

Ramez published a thread arguing the most important AI graphs — measures of how long tasks AIs can complete autonomously — differ dramatically from real-world data inside OpenAI and Google.

Ethan Mollick responds: tracking AI autonomy is critical and this is a useful counter to the most audacious claims about autonomous AI research. But focusing on research tasks at frontier labs risks underestimating what AI already does on economically valuable work. Most work is not frontier AI research: Epoch's MirrorCode results show AI autonomously reimplementing software researchers estimate would take humans 2-17 weeks — well-specified work with very different boundary conditions. He also notes some human intervention in real-world settings is fine and often preferable.

Related event: Ramez Naam Argues METR Task-Length Benchmark Overstays Real AI Capability by Up to 114x(6 posts)→

Original post →

More from AGI Musings

AGI Musings channel →