Forecast-reality gap in AI task lengths hits 165x and widens ~30x over 12 months
GaryMarcus · x · 2026-11-21
Gary Marcus retweets Ramez Naam's analysis: forecasts based on benchmarks like METR/ECI show a 100-fold gap versus real-world OpenAI task lengths, and the gap is widening — projected to grow 30x over 12 months to 3,000x. The chart shows the ECI→METR forecast gap with OpenAI task lengths grew 3.1x since June: from 53x on June 15 to 165x on October 9 (133x/177x for OpenAI/Google on Sept 1). Recent benchmark trends double every 1.7 months, while real tasks are just 8.0–16.2 minutes (OpenAI) and 9.0 minutes (Google).
Related event: Ramez Naam Claims METR Benchmark Overstates AI Task Lengths by Over 100x(5 posts)→
More from AGI Musings
- Ramez Naam extrapolation: AI a decade away from completing 98% of hour-long human tasks — GaryMarcus · 2026-11-21
- Shermer doubles down: AI means protopia, not utopia or dystopia — intellectronica · 2026-11-21
- Hassabis predicted 2 years for AI to crack a Millennium Prize problem; 6 months later, both Riemann and Navier–Stokes have proposed solutions — Hesamation · 2026-11-21
- Terence Tao on AI math: 'strip mining' that wins proofs but teaches us nothing — yacineMTB · 2026-11-21
- Imagine P=NP: Verifiable Tasks Solved by Compute, Security in Tatters — gleech · 2026-11-21
- Anthropic claims Claude proved the Riemann hypothesis; Stanford posts all-human rival proof hours later — The Decoder · 2026-11-21