Researcher Argues Task-Horizon Measure Is Incoherent: p50/p99 Greatly Overestimated
davidmanheim · x · 2026-11-11
David Manheim argues that task horizon length is incoherent as a capability measure, with a technical case:
- p99 and p50 are intrinsically connected since both are estimated from failure rates
- As reliability increases, the p50/p99 gap decreases; but due to measurement error, success rates on shorter tasks reach a maximum below 100%
- The slope of the estimated line therefore greatly overestimates p50/p99 — partly why only p80 is reported
A notable methodological challenge to the popular 'AI task horizon doubles every few months' narrative.
More from Research
- MIT and NYU's FLEX lets robots generalize a single spoken correction into broad preferences — JafarNajafov · 2026-11-11
- MIT's WorldParticle uses GPT-style next-step prediction to simulate sand, water and protein folding — thisdudelikesAI · 2026-11-11
- Exhaustive subset analysis shows a few neurons suffice to flip LLM answers, not the whole circuit — Ömer Tekin Yavuz · 2026-11-11
- Reddit debates China's ML paper output: 100–200 papers per grad student, one paper a week — Strong_Boy_757 · 2026-11-11
- DeepLearning.AI Launches Free Short Course on Tabular Foundation Models for Data Science Agents — DeepLearningAI · 2026-11-11
- Google DeepMind Weather is hiring Student Researchers for next year — LucaAmb · 2026-11-11