Ramez Naam Claims METR Benchmark Overstates AI Task Lengths by Over 100x
On November 21, Ramez Naam published a systematic critique of the industry's most important AI capability metric—the METR task-length benchmark and its extrapolations—arguing that it is severely disconnected from real-world data. Shared widely by Gary Marcus and others, the analysis sparked broad discussion.
Confirmed
- At an 80% success rate, ECI→METR estimates are 33x the actual internal task lengths at OpenAI and 41x at Google; the worst overestimate reaches 114x.
- The well-known AI 2027 fast takeoff forecast was also found to be inflated: it predicted OpenAI task lengths of 30 hours (80% success rate) on October 9, 2026, while the actual measurement was just 16.2 minutes—an overestimate of 111x.
- The gap between predictions and measurements keeps widening: the current roughly 100x divide grows by about 30x per month at the current pace, potentially reaching around 3000x within 12 months.
- Naam's extrapolation is based on the Epoch Capabilities Index growing 18.9 points per year, drawing on eight months of real OpenAI research task data from March–October 2026, and concludes it will take roughly a decade for AI to reach tasks at 98% of human task lengths.
Why it matters
- The METR task-length curve underpins AI capability forecasting and fast-takeoff scenarios like AI 2027; if its extrapolations systematically overstate real capability by two orders of magnitude, accelerationist narratives and policy judgments built on them need recalibration.
- Commentators including Gary Marcus used this to further question the industry's prevailing optimistic forecasts, arguing that the ever-widening gap between predictions and measurements suggests current capability benchmarks may be badly distorted.
2026-11-21 ~ 2026-11-21 · 5 related posts
- Episode 1: METR Quarterly Update: AI R&D Capabilities Double Every 3.6 Months and Are Getting Harder to Measure(2026-11-21, 8 posts)
- Episode 2: Ramez Naam Claims METR Benchmark Overstates AI Task Lengths by Over 100x(2026-11-21, 5 posts)
- The most important charts in AI are wrong: real task horizons 33–114x shorter than forecasts — paulnovosad · 2026-11-21
- METR task-length benchmark challenged: estimates 33-41x OpenAI and Google internal data — teortaxesTex · 2026-11-21
- AI 2027 forecast overestimates OpenAI's real-world task lengths by 111x, analysis shows — GaryMarcus · 2026-11-21
- Forecast-reality gap in AI task lengths hits 165x and widens ~30x over 12 months — GaryMarcus · 2026-11-21
- Ramez Naam extrapolation: AI a decade away from completing 98% of hour-long human tasks — GaryMarcus · 2026-11-21