METR: AI R&D capabilities rapidly improving and getting harder to measure, up to Opus 5.5 and GPT-6 Astra

najoungkim · x · 2026-11-21

METR's Q4 capability update evaluates public models up to Opus 5.5 and GPT-6 Astra on AI R&D-relevant tasks, concluding that these capabilities are improving rapidly while becoming harder to measure. The report includes time-horizon diagnostics, MCI modeling, inference scaling, swarm experiments, reward hacking analysis, and extensive appendices. The authors remain significantly uncertain about which capabilities are practically necessary for drastic R&D acceleration and how to measure them reliably. The tweet adds that evaluations used Slay the Spire (with only the Ironclad character tested).

Related event: METR: AI R&D capabilities soaring beyond measurable benchmarks(4 posts)→

Original post →

More from AGI Musings

AGI Musings channel →