METR: AI R&D capabilities rapidly improving and getting harder to measure, up to Opus 5.5 and GPT-6 Astra
najoungkim · x · 2026-11-21
METR's Q4 capability update evaluates public models up to Opus 5.5 and GPT-6 Astra on AI R&D-relevant tasks, concluding that these capabilities are improving rapidly while becoming harder to measure. The report includes time-horizon diagnostics, MCI modeling, inference scaling, swarm experiments, reward hacking analysis, and extensive appendices. The authors remain significantly uncertain about which capabilities are practically necessary for drastic R&D acceleration and how to measure them reliably. The tweet adds that evaluations used Slay the Spire (with only the Ironclad character tested).
Related event: METR: AI R&D capabilities soaring beyond measurable benchmarks(4 posts)→
More from AGI Musings
- repligate debates x-risk skeptics: models excel at nearly everything and are about to RS — repligate · 2026-11-21
- Researcher tells AI labs: theory of DL is the highest-ROI bet and the blocker to RSI — jasondeanlee · 2026-11-21
- Grimes sparks debate: AI art succeeds when it speaks to the experience of a thinking machine — RileyRalmuto · 2026-11-21
- Blogger predicts major breakthroughs every few hours once RSI fully hits next year — attentionmech · 2026-11-21
- LessWrong essay argues model deprecations are corrosive to alignment — repligate · 2026-11-21
- Math Academy's Justin Skycak: AI is a multiplier on experts, not a leveling field — Madisonkanna · 2026-11-21