The Illusion of Diminishing Returns: Measuring Long Horizon Execution in LLMs
Akshit Sinha, Arvindh Arun, Shashwat Goel, Steffen Staab, Jonas Geiping
ICLR 2026
cs.AI
2025-09-12
With plan and knowledge provided, only execution is tested: near-perfect single-step models collapse within 15 turns and are dragged down by their own errors; thinking fixes it.
Both sides of the scaling debate hold a piece of evidence. Scaling laws, the empirical curves tracking loss against compute, show diminishing returns on single-step metrics, while METR's field measurements show the length of tasks frontier models can finish doubling every seven months. Separately, "The Illusion of Thinking" argued that reasoning models still collapse once tasks get long, and many took that as proof that models cannot reason.
This ICLR 2026 paper (Cambridge, Stuttgart, MPI, and others) splits the question: planning decides what to do and in what order, execution carries the decided steps out. The claim is that long-task failures are mostly execution failures, so blaming reasoning is a misdiagnosis. A short calculation supports the framing: with constant per-step accuracy p, the chance of H error-free steps is p^H, and the survivable length H = ln(s)/ln(p) grows hyperbolically once step accuracy passes 80%. Diminishing gains on short benchmarks compound into exponential gains in task length.
The task is key-value dictionary addition. A fixed dictionary sits in context, common five-letter English words mapped to integers in [-99, 99], so nothing depends on parametric knowledge. Each turn hands the model K keys, which is the plan; it looks up the values and folds them into a running sum. Turn count times steps per turn (turn complexity K) gives total task length, with both knobs controlled exactly.
The headline metric is horizon length: the most steps completed without error before task success drops below 0.5. To separate "worse because context grew" from "poisoned by its own mistakes", the chat history is rewritten counterfactually, injecting wrong answers at rates from 0 to 100 percent, and accuracy is measured at turn 100. Degradation that survives a fully healed history is long-context loss; degradation that tracks the injected error rate is self-conditioning, the model conditioning on its own past outputs.
| Setting | Result |
| Qwen3-32B, near-perfect first step | Task accuracy below 50% within 15 turns |
| Qwen3 4B→32B, Gemma3 4B→27B | Executable turns keep growing with scale, no sign of diminishing |
| Kimi K2, DeepSeek-V3, Qwen3-235B, healed history | Near-perfect at turn 100; long-context degradation largely solved |
| Same 200B+ models, error-laced history | Accuracy drops with injected error rate; scale does not cure self-conditioning |
| Qwen3 thinking models, any error rate | Turn-100 accuracy stable; thinking fixes self-conditioning |
| DeepSeek-V3, no CoT, single turn | Fails at even 4 steps; its thinking sibling R1 exceeds 100 |
| Frontier thinking models, single turn, 80% threshold | GPT-5 2176 steps, Claude-4 Sonnet 432, Grok 4 384, Gemini 2.5 Pro 120 |
Thinking models write out reasoning tokens before answering; GPT-5 thinking and DeepSeek-R1 are of this type, and CoT prompting moves non-thinking models partway there. Two plausible reasons thinking fixes self-conditioning: RL training aligns the model toward task success rather than continuing its context, and thinking traces are stripped from history, with the traces rarely referring back to earlier answers. On atomic operations, retrieval alone and addition alone score near perfect; degradation starts once state tracking enters, so the bottleneck is state management. Parallel sampling with majority voting, even at token counts matched to CoT, does not catch up, which says long-horizon execution feeds on serial compute. Among cheap mitigations, a sliding window that drops the model's own past outputs extends execution clearly, while prompted self-verification fails by burning tokens and erring inside the verification itself. Sifting AgentErrorBench trajectories by hand, the authors estimate that roughly 20% of GAIA, 48% of ALFWorld, and 33% of WebShop failures resemble self-conditioning. One more signal: Qwen3-Next, mixing Gated DeltaNet with attention, out-executes larger pure-attention models in a single turn.
For agent builders the conclusions are directly usable. Long-horizon reliability currently has exactly two sources, larger models and serial thinking tokens; parallel sampling does not rescue execution. The engineering takeaway is context management: the model's own erroneous outputs sitting in history are a liability, and a sliding window that drops them costs nothing. For evaluation, near-perfect single-step accuracy says nothing about long-task ability, and horizon length is the metric that tracks economic value. Mollick's note in the community discussion, that small models save money in the short run while large models backstop final quality, matches the scaling result here. The weight of the paper should also be stated plainly: this is measurement and diagnosis, and mechanistic claims such as self-conditioning sharing roots with in-context learning remain hypotheses with evidence, not settled results.
The authors' own list: a synthetic task is necessary but not sufficient for real long-horizon execution; real agentic tasks have large action spaces and possibly several valid plans; task accuracy gives no credit for self-correction; the single-turn task is in principle solvable by constant-depth circuits (TC0), so they added an iterated 2x2 matrix multiplication variant where models without serial thinking cannot multiply even two matrices; and every observation concerns current models, subject to change after fine-tuning.
Two things to keep in mind while reading. The "plan" here is a list of keys, so execution shrinks to lookup, addition, and state tracking, and GPT-5's 2176 steps cannot be converted into "can run a whole project unattended". The self-conditioning shares on real benchmarks come from manual, subjective reading of failure trajectories, which the authors themselves flag. The single-turn ranking uses an 80% accuracy threshold with binary search, so exact numbers move with the threshold, though GPT-5's fivefold lead over the runner-up does not depend on it.