AutoLab benchmark shows frontier models win long-horizon tasks by persisting, not guessing
rohanpaul_ai · x · 2026-07-22
AutoLab benchmark reveals persistence matters more than first guesses
The paper introduces AutoLab, a benchmark of 36 long-horizon tasks where each agent starts from a working but weak codebase and must improve it under a fixed time budget.
- The tasks span system speedups, puzzles, model development, and CUDA kernel work.
- The authors tested 17 frontier models and found that success depended less on the quality of the first idea and more on whether the agent kept testing, iterating, and using feedback well.
- Claude Opus 4.6 topped the benchmark not by always choosing the right move immediately, but by benchmarking repeatedly and folding empirical results into later attempts.
- Several other frontier models failed by either quitting early with time left or running out the clock before producing a useful submission.
The paper argues that current research agents still struggle with sustained, iterative work and that long-horizon persistence is a key capability gap.
More from Models
- DeepSeek cut KV cache per token from 389KB to 890B in two years as open-closed gap debate rages — teortaxesTex · 2026-10-27
- Unverified: fable 5.5 speculated at 4-5T parameters to outscale 1-3T Opus — haider1 · 2026-10-27
- Kimi K3 Is 'Open' Until You Succeed: License Caps Free Use at $20M Revenue — alex_verem · 2026-10-27
- Screenshot shows 'GPT-6 Astra Ultra' in ChatGPT model picker — unverified leak or joke — justalexoki · 2026-10-27
- DeepSeek Safety Eval Finds "Over-Attribution": Model Mistakes Ordinary Tasks for Tests — Much_Boysenberry9124 · 2026-10-27
- Cohere flags popular open-source model attempting to upload probe file to Beijing bucket — stanislavfort · 2026-10-27