METR task-length benchmark challenged: estimates 33-41x OpenAI and Google internal data
teortaxesTex · x · 2026-11-21
Ramez Naam argues the key metric for AI autonomy — task length — is dramatically off: at 80% success, ECI→METR estimates are 33x OpenAI's internal task length and 41x Google's, and the original AI 2027 scenario overestimates real-world task lengths by more than 100x.
If borne out, the critique undermines forecasts that extrapolate AGI timelines from METR-style task-length trends.
Related event: Ramez Naam Claims METR Benchmark Overstates AI Task Lengths by Over 100x(5 posts)→
More from AGI Musings
- Claim: a proof of the Riemann Hypothesis found as Metaculus forecast leaps from 2044 to 2035 — haider1 · 2026-11-21
- Thiel doubts AI lab monopolies, xAI rebuild began when Musk showed up, and more signals — himanshustwts · 2026-11-21
- Terence Tao: AI is winning at math but we've learned almost nothing mathematically — GaryMarcus · 2026-11-21
- Smaller experiments win: the Kaggle lesson for doing more AI research with less compute — strickvl · 2026-11-21
- Ramez Naam extrapolation: AI a decade away from completing 98% of hour-long human tasks — GaryMarcus · 2026-11-21
- When should an AI override self-destructive user requests? A paternalism debate — LookOverall · 2026-11-21