With high-context human baselines, METR time horizons drop ~10x
BasilHalperin · x · 2026-11-09
Basil Halperin flags a detail of the METR paper (Kwa et al.): time horizons are measured against "low-context" human baselines; with higher-context baselines, time horizons come out roughly 10x lower. Two puzzles follow:
- The high-to-low context ratio would have to vary a lot across tasks to destroy the metric's predictive signal, which is surprising.
- Time horizon measures AI capability, but for economic automation what matters is high-context capability, not low-context.
He says the point needs more thought and invites takes.
More from AGI Musings
- Miyazaki: Young anime artists who only study anime are 'spinning wheels' — threepointone · 2026-11-09
- Dev Headcount Down, Customer-Facing Roles 10x: The Post-LLM Job Map — kylegawley · 2026-11-09
- ryolu_ releases full Shanghai talk: when AI can do anything, do we know what we want? — ryolu_ · 2026-11-09
- Australia Sees 108,300 Redundancies in Two Months, Highest Since COVID Amid AI Push — gaganghotra_ · 2026-11-09
- Productivity gains in OpenAI's data are actually declining over time, researcher finds — austinc3301 · 2026-11-09
- Ramez: OpenAI's Productivity Gain Per ECI Point Appears to Decline Over Time — ramez · 2026-11-09