How Fast Do Agents Rot? An Empirical Study of Long-Horizon Degradation in LLM Agents for Production Decision-Making
Shubhra Mittal
physics.soc-ph, cs.AI
2026-09-01
9 models and 10,664 trajectories show success follows r^H. On ReAct tool-use every model falls from near-perfect to near-zero within 16 steps; truncating context steepens decay.
Teams that run agent orchestration in production keep seeing the same split: the system clears offline evals, then falls apart on long, dependent workflows. Kwa et al. showed that the length of tasks models can finish has been doubling across generations, but those measurements, and most agent leaderboards, still sit on short-to-medium horizons. Horizon here means the number of dependent steps that all have to succeed.
If each step succeeds independently with probability r, a task of horizon H succeeds with probability about r to the H. Short tasks look nearly solved. Long ones collapse. This study tests whether that curve is real, and whether the driver is step count or context length.
Shubhra Mittal, a Principal Software Engineer at Microsoft AI publishing independently, ran a controlled study: nine instruction-tuned models (open weights from Llama-3.2-1B to DeepSeek-V3 at 671B, plus GPT-4o-mini, Gemini-2.5-Flash-Lite, and Claude-3-Haiku), four task families, five horizons, and three context regimes. Streaming plus agentic families yield 10,664 analyzed trajectories. Temperature is fixed and seeds are public. A simulator oracle scores success, so there is no LLM-as-judge noise.
All four families are tool-using loops:
The first three families are the streaming set. The same task instance is run under three paired regimes. Natural feeds one instruction per turn with full history. Compressed keeps the same operations and turn count but shortens context with carried state and a windowed history. Padded dumps every operation for that horizon into a single turn. That split isolates step count from context length. Functional form is chosen by AIC among geometric, threshold, and linear decay.
The geometric form wins in 28 of 36 model-task cells. The rest are threshold or cliff shaped: strong models stay near-perfect until a critical horizon, then drop. No model reaches per-step reliability of 1 on any non-trivial task, so collapse at a long enough horizon is a certainty.
Decay steepness tracks task structure. On the strongest models, Refchain is flattest, Ledger sits in the middle, Cipher is steepest. That ordering is stable enough across the roster that task structure matters about as much as raw scale.
ToolQA is the sharp result. At horizon 2 most models are near ceiling; the task is not intrinsically hard. At horizon 16 every model has collapsed.
| Model | H=2 | H=4 | H=8 | H=16 |
| Llama-3.2-1B | 0.00 | 0.00 | 0.00 | 0.00 |
| Qwen2.5-7B | 1.00 | 0.07 | 0.13 | 0.33 |
| Llama-3.1-8B | 1.00 | 0.73 | 0.07 | 0.13 |
| GPT-4o-mini | 1.00 | 1.00 | 0.20 | 0.07 |
| Gemini-2.5-Flash-Lite | 0.20 | 0.53 | 0.00 | 0.00 |
| Claude-3-Haiku | 1.00 | 0.47 | 0.00 | 0.00 |
| Llama-3.3-70B | 1.00 | 0.93 | 0.67 | 0.00 |
| Qwen2.5-72B | 1.00 | 1.00 | 0.87 | 0.13 |
| DeepSeek-V3 | 1.00 | 1.00 | 0.67 | 0.27 |
Qwen2.5-72B is perfect at H=2 and H=4, which any benchmark that stops there would treat as a pass, and falls to 0.13 at H=16. GPT-4o-mini goes from 1.00 at H=4 to 0.07 at H=16. DeepSeek-V3 is the best H=16 number in the table, at 0.27. Claude-3-Haiku and Gemini-2.5-Flash-Lite are already 0 at H=8. Llama-3.2-1B is 0 from the shortest setting.
On the open-model ladder, per-step reliability rises with parameter count (Pearson r = +0.36 vs log10(params)) with diminishing returns. The hosted models sit with the strongest open ones. Nobody pushes r to 1.
A pure geometric law assumes a constant hazard. The hazard rises: mean per-step accuracy falls from 0.58 in the first third of long trajectories to 0.44 in the last third. The first error tends to land about a third of the way through, and recovery is rare. Format or tool-call drift, a turn that cannot be parsed as a valid action, hits 21% of trajectories and rises with horizon. Errors behave as absorbing states.
The driver is step count, not context length. Natural is best at every horizon. Logit slope per horizon doubling is -0.44 for natural, -0.69 for compressed (p=3×10^-6 vs natural), and -0.40 for padded (p=0.51, indistinguishable from natural). Bounding the window steepens decay, which is the opposite of a lost-in-the-middle prediction. Dumping every operation into one prompt decays at essentially the same rate as natural, just from a lower start, so multi-turn conversational structure itself is not the cause.
Projecting the measured mean r=0.61 onto typical benchmark horizons gives 0.42 at GAIA-length (8 steps), 0.36 at WebArena (15), 0.33 at τ-bench (20), 0.30 at SWE-bench / OSWorld (30), and 0.24 at hundred-step production horizons.
For people who ship agent runtimes, this turns "evals pass, production fails" into a budget. Measure per-step reliability on your own tasks. The achievable horizon is a function of that r, not of a GAIA or SWE-bench aggregate. Because the hazard accelerates, budget below the geometric estimate.
Naive context truncation, a common cost and latency tactic, backfires here. Step-level checks are more useful: an end-to-end success monitor only fires after the trajectory has already failed. Where an oracle or a cheap consistency check can be inserted, failures show up earlier and cheaper.
For benchmark designers, a single pass rate samples the flat top of a curve that falls out of frame. Reporting success against horizon would make those numbers more predictive of production. This is a measurement warning, not a new agent method.
The tasks are synthetic, by design, to get an exact oracle and a clean horizon variable. ToolQA answers the "this is not really an agent" objection, but ecological validity against fully open-ended real workflows is still open.
The ToolQA API budget ran out before the longest horizon, so the reported collapse at H=16 is a lower bound on severity, not a ceiling. Nine models across six vendor families do not prove every current or future model follows the same geometric form. Three additional small models were dropped for failing the structured-output protocol on hosted routes. Decoding temperature is fixed at a moderate setting; sensitivity was not measured.
The proprietary roster is the mini / Flash-Lite / Haiku tier, not GPT-4o, Claude Sonnet, or flagship Gemini. Reading "widely deployed systems" as frontier models overclaims the evidence.
Qwen2.5-7B on ToolQA rises from 0.07 at H=4 to 0.33 at H=16, and Gemini-2.5-Flash-Lite goes 0.20 to 0.53 then to 0. Non-monotonic cells sit awkwardly next to a geometric story. The paper uses Wilson intervals because some cells hug 0 or 1, but Table 2 does not report trials per cell, so a noisy cell can be read as a law.