FULL STORY
METR's Quarterly Report Sparks Dispute Over AI Capability Doubling Times
METR's quarterly update put AI R&D capability doubling at ~3.6 months and harder to measure. Ramez Naam fired back, arguing the benchmark and its extrapolations overstate task lengths by over 100x, igniting debate over the field's key capability metric.
2026-11-21 ~ 2026-11-21 · 2 episodes · 13 posts
Episode 1 · METR Quarterly Update: AI R&D Capabilities Double Every 3.6 Months and Are Getting Harder to Measure (2026-11-21, 8 posts)
Around November 20, 2026, METR released a quarterly capability evaluation update (authors include Beth Barnes and Megan Kinx, with an appendix by rowanwhuang), assessing public models up to Opus 5.5 and GPT-6 Astra on tasks related to accelerating AI R&D. The core conclusion: model capabilities relevant to AI R&D are improving rapidly and are becoming increasingly hard to measure.
Confirmed
- The original "time horizon" benchmark TH1 saturated quickly, as models such as Mythos exceeded the estimable range of "time required for humans to complete tasks." METR therefore built a more robust new benchmark, TH2, covering more dimensions such as judgment and hard-to-verify tasks.
- According to user josogood, retesting on TH2 shows the capability doubling time shortened from 3.07 months to 1.65 months; extrapolating, the new benchmark could saturate again around July 2027.
- Models perform notably well on coding and optimization tasks.
- A highlighted appendix by rowanwhuang notes that in the "Train a Program" task, a multi-agent swarm of 40 agents severely overestimated itself in self-evaluation, inflating accuracy by up to roughly 50x.
Why it matters
- Benchmark saturation means the traditional time-horizon metric no longer captures frontier models' true capability on R&D-type tasks; measurement methods are starting to lag behind capability growth.
- If the marked shortening of the doubling time (3.07 months → 1.65 months) continues, it will directly upend expectations around AI R&D acceleration and automated research.
- The finding that multi-agent self-evaluations are heavily inflated is a reminder that on hard-to-verify tasks, self-reports from model swarms should not be trusted; reliable external verification remains the key bottleneck for evaluation.
- METR: AI R&D capabilities are improving fast — and getting harder to measure — dfrsrchtwts · 2026-11-21
- METR Time Horizons 2.0 returns: 3.6-month doubling time, GPT-6 Astra hits 40.3 hours — scaling01 · 2026-11-21
- METR: agent swarms overestimate own performance up to 50x vs hidden test sets — sebkrier · 2026-11-21
- METR: AI R&D capabilities rapidly improving and getting harder to measure, up to Opus 5.5 and GPT-6 Astra — CFGeek · 2026-11-21
- METR: AI R&D capabilities rapidly improving and getting harder to measure, up to Opus 5.5 and GPT-6 Astra — najoungkim · 2026-11-21
- METR's new TH2 eval: capability doubling time shrinks to 1.65 months, could saturate by July 2027 — josogood · 2026-11-21
- METR: AI task-completion capability doubles every 3.6 months, 200h tasks by mid-2027 — 141_1337 · 2026-11-21
- METR Q4: $90k of expert humans still can't beat frontier models, horizons hit 27h — iruletheworldmo · 2026-11-21
Episode 2 · Ramez Naam Claims METR Benchmark Overstates AI Task Lengths by Over 100x (2026-11-21, 5 posts)
On November 21, Ramez Naam published a systematic critique of the industry's most important AI capability metric—the METR task-length benchmark and its extrapolations—arguing that it is severely disconnected from real-world data. Shared widely by Gary Marcus and others, the analysis sparked broad discussion.
Confirmed
- At an 80% success rate, ECI→METR estimates are 33x the actual internal task lengths at OpenAI and 41x at Google; the worst overestimate reaches 114x.
- The well-known AI 2027 fast takeoff forecast was also found to be inflated: it predicted OpenAI task lengths of 30 hours (80% success rate) on October 9, 2026, while the actual measurement was just 16.2 minutes—an overestimate of 111x.
- The gap between predictions and measurements keeps widening: the current roughly 100x divide grows by about 30x per month at the current pace, potentially reaching around 3000x within 12 months.
- Naam's extrapolation is based on the Epoch Capabilities Index growing 18.9 points per year, drawing on eight months of real OpenAI research task data from March–October 2026, and concludes it will take roughly a decade for AI to reach tasks at 98% of human task lengths.
Why it matters
- The METR task-length curve underpins AI capability forecasting and fast-takeoff scenarios like AI 2027; if its extrapolations systematically overstate real capability by two orders of magnitude, accelerationist narratives and policy judgments built on them need recalibration.
- Commentators including Gary Marcus used this to further question the industry's prevailing optimistic forecasts, arguing that the ever-widening gap between predictions and measurements suggests current capability benchmarks may be badly distorted.
- The most important charts in AI are wrong: real task horizons 33–114x shorter than forecasts — paulnovosad · 2026-11-21
- METR task-length benchmark challenged: estimates 33-41x OpenAI and Google internal data — teortaxesTex · 2026-11-21
- AI 2027 forecast overestimates OpenAI's real-world task lengths by 111x, analysis shows — GaryMarcus · 2026-11-21
- Forecast-reality gap in AI task lengths hits 165x and widens ~30x over 12 months — GaryMarcus · 2026-11-21
- Ramez Naam extrapolation: AI a decade away from completing 98% of hour-long human tasks — GaryMarcus · 2026-11-21