METR's Updated Time-Horizon Plot Shows Models Still Lag on Judgment Tasks
MIT Tech Review AI · rss · 2026-11-21
MIT Tech Review covers METR's major update to its widely-misread time-horizon plot, exposing the gap between verifiable skill and judgment.
Background & update
- The original plot showed the human-time length of tasks frontier models can complete doubling every 4 months, but its complexity led to widespread misinterpretation.
- The new version adds tasks estimated to take humans far longer, plus more open-ended ones.
Key findings
- Progress on verifiable tasks (coding, math) keeps compounding via reinforcement learning on automatically checkable outcomes.
- But models fall short on judgment tasks: none of the AI-designed evaluations were deemed usable by METR staff — though the latest models like Opus 5.5 reached "borderline above the bar," up from "clearly below" a year ago. Models also underperformed humans at scoring philosophical arguments.
- Experts like Sayash Kapoor suggest "research taste" may be the key bottleneck to AI R&D automation; OpenAI's same-day blog similarly admitted its models do worse on real-world tasks than on METR-style verifiable ones.
Methodological limits
- Human completion times for long tasks are mostly estimates; the philosophy task was largely rated by a single person, which could shift scores.
- Experts welcome the creative expansion of evals but caution against extrapolating trends.
Context
- A heavily redacted earlier METR report claimed AI might be speeding up capability progress at Anthropic by a factor of two, fueling recursive self-improvement fears.
More from AGI Musings
- Writer Uses Claude to Pen a Culture Short Story — and Admits It's Obviously AI-Written — NathanpmYoung · 2026-11-22
- When Creating Life Costs Less Than Caring for It: An AI Biotech Question — PierceLilholt · 2026-11-22
- Closed models, opaque processes: will AI-made discoveries only benefit an elite? — GeorgiaChal · 2026-11-22
- Contrarian Take: LLMs as the Force Multiplier for True Genius — 2a_lib · 2026-11-22
- AI Velocity Ledger catalogs 41 verified feats of machine-accelerated discovery — JosephJacks_ · 2026-11-22
- Lawyer Pushback: Passing the Bar Doesn't Mean AI Can Argue in Court Yet — TinfoilTricorn · 2026-11-22