David Krueger questions METR task-length metric: does coding progress generalize?
DavidSKrueger · x · 2026-11-09
AI safety researcher David Krueger questions METR's task-length/RSI measures: little serious research tests whether coding/math progress generalizes to other tasks. He expects an "accuracy on the line" effect (OOD performance correlated with the metric) but the slope is unknown, and RL-based gains may generalize differently than SL. Accelerating AI R&D aids some form of RSI but might yield a savant rather than AGI. He laments that SOTA-lab research is untrustworthy due to incentives and independent research lags the frontier, yet still backs independent research. Chart: fMoW-WILDs showing ID-vs-OOD accuracy correlation with slope <1.
Related event: METR's 1.9x AI Speedup Claim Sparks Debate Over Benchmark Fitting Methods(17 posts)→
More from AGI Musings
- Owain Evans flags Daniel Kokotajlo's new LessWrong shortform as worth reading — OwainEvans_UK · 2026-11-09
- PNAS paper compares children, adults and AI agents exploring the same open-ended environment — AlisonGopnik · 2026-11-09
- Designer warns AI tools strip out struggle needed to build judgment, calls for slower tools — jakubzeg · 2026-11-09
- Free 1-hour AI lecture: 95% of people still use AI like a calculator — LinusEkenstam · 2026-11-09
- US business services added 70,000 jobs in October despite AI exposure fears — kimmonismus · 2026-11-09
- AI timelines debates go nowhere because everyone secretly defines superintelligence differently — sebkrier · 2026-11-09