Forecast-reality gap in AI task lengths hits 165x and widens ~30x over 12 months

GaryMarcus · x · 2026-11-21

Gary Marcus retweets Ramez Naam's analysis: forecasts based on benchmarks like METR/ECI show a 100-fold gap versus real-world OpenAI task lengths, and the gap is widening — projected to grow 30x over 12 months to 3,000x. The chart shows the ECI→METR forecast gap with OpenAI task lengths grew 3.1x since June: from 53x on June 15 to 165x on October 9 (133x/177x for OpenAI/Google on Sept 1). Recent benchmark trends double every 1.7 months, while real tasks are just 8.0–16.2 minutes (OpenAI) and 9.0 minutes (Google).

Related event: Ramez Naam Claims METR Benchmark Overstates AI Task Lengths by Over 100x(5 posts)→

Original post →

More from AGI Musings

AGI Musings channel →