AI autonomy benchmarks may be wrong, says Ramez; Mollick adds caveats
eldonredwards · x · 2026-11-21
Ramez published a thread arguing the most important AI graphs — measures of how long tasks AIs can complete autonomously — differ dramatically from real-world data inside OpenAI and Google.
Ethan Mollick responds: tracking AI autonomy is critical and this is a useful counter to the most audacious claims about autonomous AI research. But focusing on research tasks at frontier labs risks underestimating what AI already does on economically valuable work. Most work is not frontier AI research: Epoch's MirrorCode results show AI autonomously reimplementing software researchers estimate would take humans 2-17 weeks — well-specified work with very different boundary conditions. He also notes some human intervention in real-world settings is fine and often preferable.
More from AGI Musings
- Critique of AI Lean proofs: formalization matters, labs leave verification to unpaid community — gerardsans · 2026-11-21
- Counter to LeCun: LLMs May Skip Human-Level and Deliver Superhuman Intelligence — hbouammar · 2026-11-21
- Yoav Goldberg: many harness abilities shift from essential to marginally useful — yoavgo · 2026-11-21
- 'Frictional AI': Adding Resistance to AI to Prevent Human Skill Decay — JeremyNguyenPhD · 2026-11-21
- AI Is Testing What Economists Know About Bubbles — bloomberg · 2026-11-21
- Do executives owe any duty to future shareholders before AI's shoe drops? — AaronBergman18 · 2026-11-21