Debate rages over METR's 1.9x AI acceleration finding: fitting methods and withheld evidence in focus

On November 8, researcher JacquesThibs posted a series of threads on METR's benchmark-fitting methodology, debating with Jeff Ladish, bshlgrs, and others over the conclusion that "the time horizon of AI tasks on the AECI benchmark halves roughly every 7 months (a 2x speedup)." The core dispute is whether the report actually tested and ruled out the capability "jump" hypothesis.

Confirmed

• JacquesThibs noted that the hypothesis METR rejected was "a one-time jump followed by a return to the original trend (slope unchanged after the jump)" — also the hypothesis in Anthropic's system card appendix; Toby Ord's model, however, is not a return-to-trend one, and METR did not test hybrid models where the slope changes after a jump.
• He challenged the report's use of piecewise-linear fits with no discontinuous jumps, which by construction sets discrete capability jumps to zero instead of letting the data estimate them. He also argued that independent evidence, without relying on training compute differences between the two model groups, already supports the idea that larger-scale pretraining brings capability leaps.
• On the 2x speedup figure, he believes it may still hold within error margins, but only under different fitting assumptions for the AECI benchmark — implying METR must have more information than the report discloses (i.e., the conclusion may depend on undisclosed private information).
• Jeff Ladish thinks Toby Ord is wrong but welcomes him defending his position; the two sides went through multiple rounds of back-and-forth on the dispute.

Unconfirmed

• Whether undisclosed data or fitting details underlie the METR report, and whether the 2x speedup conclusion is robust under alternative fitting assumptions, remains unsettled.
• Whether the jump hypothesis or the continuous-trend hypothesis fits the data better is still being debated.

Why it matters

• The dispute directly bears on the credibility of the widely cited "AI capabilities double every 7 months" conclusion: if it depends on particular fitting choices or undisclosed information, its extrapolations (such as future capability timelines) should be treated more cautiously.
• The debate also highlights the risk of prejudiced model specification in benchmark fitting: excluding discrete jumps by construction may systematically underestimate the possibility of capability leaps, with implications for both safety assessment and trend forecasting.

2026-11-08 ~ 2026-11-08 · 14 related posts

Full story(5 episodes)→

• Episode 1: METR's New Capability Chart Sparks Debate Over Leaked AECI Benchmarks(2026-11-06, 2 posts)
• Episode 2: METR Defends Anthropic Evaluation Deal as Better Than Self-Assessment(2026-11-07, 2 posts)
• Episode 3: METR capability-doubling report stirs panic and pushback in AI safety circles(2026-11-07, 7 posts)
• Episode 4: METR evaluation report sparks transparency and conflict-of-interest concerns(2026-11-07, 4 posts)
• Episode 5: Debate rages over METR's 1.9x AI acceleration finding: fitting methods and withheld evidence in focus(2026-11-08, 14 posts)

• Researcher flags scaling fit that hard-codes capability jump to zero — JacquesThibs · 2026-11-08
• METR finds Anthropic model progress accelerating 1.9x since Oct 2025; Hanson calls it one-time speedup — sebkrier · 2026-11-08
• Debate over METR benchmark fit: whether the 1.9x slope increase holds up — JacquesThibs · 2026-11-08
• [source] Debate over METR's 2x speedup claim: does it lean on private information? — JacquesThibs · 2026-11-08
• [source] Researcher disputes METR report reading: rejected hypothesis isn't Toby Ord's mixed model — JacquesThibs · 2026-11-08
• METR rejected the jump-with-same-slope hypothesis, not Toby's mixed model, argues researcher — JacquesThibs · 2026-11-08
• Debate over METR's extrapolation report: which Toby Ord hypothesis was actually rejected — bshlgrs · 2026-11-08
• Following the METR debate: arguing evidence is weak backfires via trust in researchers — bshlgrs · 2026-11-08
• Debate: METR's 2x capability extrapolation may have missed the jump-plus-slope model — JacquesThibs · 2026-11-08
• Independent analysis cross-checks METR report: Anthropic internal models show ~1.9x capability acceleration — RishiBommasani · 2026-11-08
• Toby Ord: correlated benchmark-ceiling errors weaken evidence for AECI slope increase — tobyordoxford · 2026-11-08
• [source] METR sensitivity analysis: AI capability slope increase often vanishes by method — tobyordoxford · 2026-11-08
• Toby Ord: priors favor level jump from pre-training, slope gain from AI R&D — tobyordoxford · 2026-11-08
• Toby Ord: scarce public evidence limits METR-style capability trend analysis — tobyordoxford · 2026-11-08