METR: AI R&D capabilities are improving fast — and getting harder to measure
dfrsrchtwts · x · 2026-11-21
METR published a November 2026 research update: AI R&D-relevant capabilities are improving rapidly and getting harder to measure, beyond what its prior time-horizon tasks captured.
- Models usually match or outperform skilled humans on coding and optimization tasks
- Weak spots: models do somewhat worse on costlier tasks (though improvement after RL training environments seems similar in rate), and show worse judgment than humans on tasks without empirical feedback
- Methodology changes: TH2 subsets split into verifiable / hard-to-verify / judgment tasks, multi-threshold scoring, human or model judges, estimated baselines
- Steady trends across the public frontier (tested up to Opus 5.5 / GPT-6 Astra), plus uncertain evidence of developer-specific trend changes; later internal models not measured
- Open questions: which capabilities are actually necessary for AI R&D, and how fast are cheap-to-verify vs expensive-to-verify capabilities improving?
More from AGI Musings
- Taking AI Seriously: Borretti traces how programmer AI advice keeps retreating — zetalyrae · 2026-11-21
- Sundar Pichai memes Euler and Gauss gawking at a 'Riemann Hypothesis Proved' headline — sundarpichai · 2026-11-21
- Blogger argues starving AI of capital is the real way to stop the singularity — JensHonack · 2026-11-21
- Delip Rao: a human RH proof is also under peer review, submitted two months ago — deliprao · 2026-11-21
- Their dangerous capability overhang, our blessed first mover advantage — nptacek · 2026-11-21
- Don't anthropomorphize LLMs: talk data coverage and error rates instead — gerardsans · 2026-11-21