Measuring AI Ability to Complete Long Software Tasks
Thomas Kwa, Ben West, Joel Becker, Amy Deng, Katharyn Garcia, Max Hasin, Sami Jawhar, Megan Kinniment, Nate Rush, Sydney Von Arx, Ryan Bloom, Thomas Broadley, Haoxing Du, Brian Goodrich, Nikola Jurkovic, Luke Harold Miles, Seraphina Nix, Tao Lin, Chris Painter, Neev Parikh, David Rein, Lucas Jun Koba Sato, Hjalmar Wijk, Daniel M. Ziegler, Elizabeth Barnes, Lawrence Chan
NeurIPS 2025
cs.AI, cs.LG
2025-03-19
METR proposes '50% task-completion time horizon': tasks humans finish that AI completes with 50% success. Frontier models now reach ~50 minutes, doubling every 7 months since 2019.
Benchmark scores keep climbing, but what does a 3-point MMLU gain mean for someone actually deploying an agent? Nobody can say. METR (Model Evaluation & Threat Research) reframes the question in plain terms: how long a task can today's AI handle on its own?
They measure capability in units of time. Instead of asking "what score does the model get," they ask "what is the longest task that AI completes with 50% success, where humans typically take that long to finish it?" This is the 50%-task-completion time horizon. The 50% mark is a compromise: 95% is too data-hungry, a lower bar says too little.
To make the duration credible they timed humans directly: domain experts (about 5 years of experience on average) actually did each task, and the dataset accumulated 800-plus human baselines totaling 2,529 hours.
The task suite combines three sources: 97 HCAST software tasks (1 minute to 30 hours), 7 RE-Bench ML-research engineering tasks (8 hours each), and 66 new "software atomic actions" (SWAA, single steps of 1 to 30 seconds). Together, 170 tasks spanning seconds to hours.
Models run as agents with tools, code edits, and error logs, close to real use. Each model-task pair runs about 8 times for a success rate, mapped back to the median human time for that task to read off the 50% horizon.
They tested 12 frontier and 4 near-frontier models from 2019 to 2025, from GPT-2 to o3 and Claude 3.7 Sonnet.
Plotting each model's 50% horizon against release date yields a near-straight exponential:
| Model | 50% time horizon |
| GPT-2 (2019) | 2 seconds |
| o3 (2025) | 110 minutes |
Claude 3.7 Sonnet and o3 complete several tasks over 4 hours. The doubling period is about 207 days (95% CI 166 to 240), roughly 7 months. Growth from 2023 to 2025 ran about 20% faster than the 2019 to 2025 average, suggesting acceleration.
The 80% horizon is 4 to 6x shorter than the 50% one: raising the bar from "barely succeeds" to "reliably succeeds" costs an order of magnitude. Inter-task agreement on which tasks models solve sits around 0.73.
The chart gives practitioners a comparable ruler. A deployment need like "my agent must run unattended for about 2 hours" can now be read directly off the curve.
The bigger claim is the extrapolation: if the exponential holds, within 5 years (the paper's window: mid-2028 to mid-2031) models will complete software tasks that take humans about a month (167 work hours). That is the planning horizon for deployment and governance.
The authors are candid. These are structured software tasks, not representative of messier real work; absolute values could be off by up to 10x; the data is too thin to nail down a 95% bar. The 5-year extrapolation assumes the exponential survives real-world tasks, which nobody can guarantee.
The drivers of growth deserve scrutiny: the paper attributes it to better error recovery, tool use, and logical reasoning, but these are qualitative judgments, not a causal decomposition.