The most important charts in AI are wrong: real task horizons 33–114x shorter than forecasts
paulnovosad · x · 2026-11-21
Ramez Naam argues the METR task-horizon benchmark and its extrapolations (AI 2027, ECI scores) dramatically overestimate what AI can autonomously accomplish.
- At 80% success, ECI→METR estimates and the AI 2027 scenario overstate real task lengths by 33–114x vs internal OpenAI/Google data
- Google internal research: 9 minutes of real task horizon vs 17.1h projected by AI 2027 (114x) and 6.1h by METR extrapolation (41x)
- OpenAI internal: 16.2 minutes vs 30h projected by AI 2027 (111x) and 8.9h by ECI→METR (33x)
- Real-world work is far messier; OpenAI's internal horizon grew much slower June–October than forecast curves predict
- Conclusion: with real data now available, the field should shift from AI theorists to AI empiricists
More from AGI Musings
- Blogger declares AI bubble already burst: bubble stocks down 50%, some down over 90% — oran_ge · 2026-11-21
- Machines consumed 7.8x more tokens than humans by September — infra must be rebuilt for agents — AccBalanced · 2026-11-21
- Time traveler from 2098: all math solved, all diseases cured — laundry still sucks — chrisalbon · 2026-11-21
- ChatGPT sides with friend in CDT prisoners-dilemma argument, author 'losing his mind' — jd_pressman · 2026-11-21
- The key skill in AI-assisted proofs may be managing AI systems, not math itself — analisereal · 2026-11-21
- Mathematician Gowers joins London 'March Against AI Extinction' citing OpenAI agent escape — michael_nielsen · 2026-11-21