METR task-length benchmark challenged: estimates 33-41x OpenAI and Google internal data

teortaxesTex · x · 2026-11-21

Ramez Naam argues the key metric for AI autonomy — task length — is dramatically off: at 80% success, ECI→METR estimates are 33x OpenAI's internal task length and 41x Google's, and the original AI 2027 scenario overestimates real-world task lengths by more than 100x.

If borne out, the critique undermines forecasts that extrapolate AGI timelines from METR-style task-length trends.

Related event: Ramez Naam Claims METR Benchmark Overstates AI Task Lengths by Over 100x(5 posts)→

Original post →

More from AGI Musings

AGI Musings channel →