METR-style fit shows benchmark gains accelerating 1.9x, but sensitive to measurement
jessi_cata · x · 2026-11-06
jessicata discusses a METR-style AECI analysis: including Anthropic's internal models, benchmark scores as of Oct 2026 are growing at 1.9x their pre-Oct 2025 rate (95% CI 1.5x–2.8x), with the trend slope jumping from +15/yr to +29/yr. But she stresses the effect is statistically sensitive — "most" measurement choices yield under 1.8x. Beth May Barnes notes the paper's appendices detail what drives the gap vs public ECI estimates (benchmark difficulty and composition assumptions), and the authors caution the fit could overstate acceleration if Anthropic's historical progress was unusually low. A rigorous debate on whether AI progress is genuinely accelerating or an artifact of measurement.
More from Models
- Zhipu chief scientist reveals GLM pretraining data roadmap: GLM-5 at 28.5T tokens, 100T next — rickasaurus · 2026-11-06
- Hesamation: OpenAI Models Beat Anthropic's Only at Certain Kinds of Games — Hesamation · 2026-11-06
- Meta's new 'walrus' checkpoint reportedly matches crouton-alpha quality — ChrisGPT · 2026-11-06
- Quick test: Gemini 3.1 Pro knows the answer in its weights, Flash doesn't — giffmana · 2026-11-06
- Ripple-65B flops in single-GPU coding test against Qwen 27B despite SOTA claims — DZhang50 · 2026-11-06
- Ripple-65B open-weight model runs at 7 tok/s locally on a 64GB Mac mini — DZhang50 · 2026-11-06