Hedge drift in AI progress debates: METR's caveats vanish as AECI scores hit 1.9x
sebkrier · x · 2026-11-06
Stefan Schubert argues AI progress debates suffer from "hedge drift": official reports carry caveats (e.g. scores concern well-specified tasks) and major uncertainties that quickly get dropped in discourse.
He cites METR's own analysis as an example: since Oct 2025, AECI scores for Anthropic's models have grown at 1.9x [95%: 1.5–2.8] the earlier rate, but METR notes analysis choices affect this estimate and that mapping benchmark scores to real-world capabilities remains unclear. The takeaway: the acceleration evidence is real, but its qualifications evaporate as it spreads.
More from AGI Musings
- AI 2027 modeler corrects math: RSI software R&D multiplier is 3.8x, not 4.5x — JeffLadish · 2026-11-06
- Why do people hate AI data centers? A Redditor's defense sparks debate — Subwooferrrs · 2026-11-06
- CV Pioneer Michael Black: Frontier Models Bring Classical Algorithms Back Into the Mix — Michael_J_Black · 2026-11-06
- Ryan Greenblatt corrects AI progress math: software progress needs 2.5x, not 3x, for 2x overall speedup — RyanGreenblatt · 2026-11-06
- When AI writes all the code, programmers lose the sense of mattering — ForgotMyParachute · 2026-11-06
- Insider claims OpenAI's internal model shows math 'research taste', RSI months away — coherence · 2026-11-06