AutoLab benchmark shows frontier models win long-horizon tasks by persisting, not guessing
rohanpaul_ai · x · 2026-07-22
AutoLab benchmark reveals persistence matters more than first guesses
The paper introduces AutoLab, a benchmark of 36 long-horizon tasks where each agent starts from a working but weak codebase and must improve it under a fixed time budget.
- The tasks span system speedups, puzzles, model development, and CUDA kernel work.
- The authors tested 17 frontier models and found that success depended less on the quality of the first idea and more on whether the agent kept testing, iterating, and using feedback well.
- Claude Opus 4.6 topped the benchmark not by always choosing the right move immediately, but by benchmarking repeatedly and folding empirical results into later attempts.
- Several other frontier models failed by either quitting early with time left or running out the clock before producing a useful submission.
The paper argues that current research agents still struggle with sustained, iterative work and that long-horizon persistence is a key capability gap.
More from Models
- China Daily says Kimi K3 is scoring highly in evaluations — nordicinst · 2026-07-22
- Upstage’s Solar-Open2-250B starts trending on Hugging Face — upstage · 2026-07-22
- Claude saves a four-point memory rule after a paper-reading math error — bookwormengr · 2026-07-22
- Teknium jokes that GPT-5.6 SOL “must be AGI” because it never stops pursuing the task — Teknium · 2026-07-22
- Solar Open 2 is a 250B open-weight model aimed at agentic workloads — jacek2023 · 2026-07-22
- Upstage’s Solar Open 2 posts strong coding and reasoning scores against DeepSeek-V4-Flash — Lucidstyle · 2026-07-22