SlopCodeBench: Benchmarking How Coding Agents Degrade Over Long-Horizon Iterative Tasks
Gabriel Orlanski, Devjeet Roy, Alexander Yun, Changho Shin, Alex Gu, Albert Ge, Dyah Adila, Nicholas Roberts, Frederic Sala, Aws Albarghouthi
www
cs.SE, cs.AI, cs.CL
2026-03-26
Fifteen coding agents extend their own solutions across 36 problems and 196 checkpoints. None finish; best strict pass rate is 14.8%, and erosion rises on 77% of runs.
Most coding-agent benchmarks ask whether the current solution is correct. The spec arrives whole, or the agent patches a repository whose architecture someone else already chose. The costly failure comes later. Hardcoding one branch per language works until the next checkpoint adds three languages and the structure has to be rebuilt. Pass rate can still look fine after the code has become hard to extend.
Recent multi-turn benchmarks often spare the agent that cost. Splitting a finished solution into dependency-ordered subproblems publishes the internal interfaces up front. Walking the commit history of a mature repository starts from an architecture humans already picked. In both setups the agent never has to live with its own first design.
SlopCodeBench, from the University of Wisconsin-Madison, Washington State University, and MIT, is 36 handwritten problems and 196 checkpoints. Specs state only observable CLI or API behavior. The agent starts from an empty directory. Each later checkpoint shows the new spec and the agent's own workspace. Prior dialogue is dropped. The tests stay hidden.
Every checkpoint adds behavior on top of the previous workspace. A reference solution never replaces the agent's code. The running example, codesearch, takes five steps: exact and regex search over Python, then JavaScript and C++, then AST patterns, then auto-fix, then Go, Rust, and Java. An implementation that hardcodes a branch per language copies that branch for every new language.
Regression tests carry old failures forward, so correctness is split three ways. Strict requires every test at the current checkpoint. The denominator is fixed at 196, and checkpoints the run never reaches count as failures. Isolated drops regressions and scores only the new behavior. Core covers behavior the spec states or shows in an example. Hidden tests also cover exhaustive functionality and error paths. If the agent crashes, later checkpoints score 0 for correctness. Erosion and verbosity are computed only where a workspace was actually produced.
Structural erosion asks whether complexity is piling into functions that are already complex. A function's mass is cyclomatic complexity times the square root of its source lines. The square root stops raw length from dominating the score. Erosion is the share of total mass held by functions whose cyclomatic complexity exceeds 10, the cutoff used by the static analyzer Radon. Verbosity is the fraction of lines flagged by one of 137 AST-grep rules or duplicated as structural clones, with a line counted once if several rules hit it. Both scores sit in [0, 1] and can move independently.
Evaluation uses each vendor's own CLI harness, not a shared loop such as SWE-agent. Each checkpoint gets a fresh container and keeps only the working directory. The limit is two hours of wall clock, with no cap on turns or spend. The default just-solve prompt only says to solve the task and manage dependencies. Problems are handwritten and reviewed by a second author. Tasks a frontier agent could finish in one shot, or that forced no design choice, were removed. The specs are language-agnostic. This paper scores Python only, because of cost.
Across 15 models from six providers, no run finishes any of the 36 problems from the first checkpoint to the last.
| Model | Strict | Isolated | Core | USD/ckpt | Erosion | Verbosity |
| GPT 5.5 | 14.8% | 28.1% | 66.8% | 1.51 | 0.49 | 0.32 |
| GPT 5.4 | 10.7% | 23.5% | 62.8% | 0.72 | 0.51 | 0.33 |
| GPT 5.3 Codex | 11.2% | 26.0% | 60.7% | 0.66 | 0.64 | 0.46 |
| Opus 4.6 | 9.7% | 20.9% | 67.3% | 3.17 | 0.75 | 0.44 |
| Opus 4.5 | 9.2% | 17.3% | 57.7% | 2.53 | 0.70 | 0.43 |
GPT 5.5 has the best strict rate, 14.8%, and the best isolated rate, 28.1%. The highest core rate in the table is 67.3% on Opus 4.6. The prose assigns that 67.3% to Opus 4.5. GPT 5.5 strictly solves at least one checkpoint on 50% of problems.
From the first phase to the last, core pass rate falls from 64.6% to 35.5%, and the gap versus isolated pass rate widens from 2.5x to 5.4x. Hidden functionality tests barely move, 62.7% to 57.7%. Error-path tests fall from 80.1% to 62.2%. Mean cost per checkpoint rises 2.2x, while the share of lines changed falls from 97.4% early to 29.5% late.
Erosion rises on 77% of trajectories and verbosity on 75.5%. Functions with cyclomatic complexity of at least 10 grow from 3.6 on average to 23.7. Mean maximum complexity grows from 27.5 to 69.0. Structural clones grow 96%. AST-grep violation density grows 0.3%. On forge, Opus 4.7 lengthens main() from 38 lines to 240 and raises its cyclomatic complexity 7x.
At HEAD, 473 Python repositories average 0.19 verbosity and 0.34 erosion. The 2,869 agent checkpoints average 0.44 and 0.68, about 2.3x and 2.0x. Only 18 repositories exceed the agent verbosity mean. Per checkpoint, agent verbosity rises by 0.0144 and erosion by 0.0264. Human medians are 0.0022 and 0.0053, so agents accumulate the two scores about 6.6x and 5.0x faster. Erosion rises in 53% of the human repositories. Among 321 repositories with at least three commits on each side of January 2024, within-repo medians move only +0.002 in verbosity and +0.014 in erosion.
An anti-slop prompt that names verbose and over-built patterns to avoid lowers GPT 5.5 erosion from 0.49 to 0.21 and verbosity from 0.32 to 0.21. Verbosity falls 27.5% on GPT 5.4 and 32.8% on GPT 5.3 Codex. A plan-first prompt, which demands a written plan before code, moves the scores less. Mean strict pass rate drops 2.4 points with anti-slop and 3.6 with plan-first. GPT 5.5 falls from 14.8% to 9.2% and 8.2%. Cost per checkpoint rises 12.1% on average. The only setting where quality improves from start to finish is GPT 5.4 with anti-slop. Reported quality velocity stays about 1.3 points per checkpoint.
A single-shot leaderboard says whether this spec can be handed in. SlopCodeBench asks whether the repository gets harder to change when tests are invisible and the architecture is the agent's own.
The prompt result is usable as is. A ban list can cut GPT 5.5 erosion from 0.49 to 0.21. The per-checkpoint slide continues, strict accuracy falls, and spend rises 12.1%.
Next to real repositories, the gap that matters is the rate of accumulation. The post-2024 shift inside human repos is much smaller than a single agent checkpoint.
Only Python is scored. Whether other languages pile logic into large functions the same way is not measured.
Both scores are proxies. The cutoff at cyclomatic complexity 10 is Radon's convention, not a threshold fit on these 36 problems. Verbosity depends on 137 handwritten rules, including anti-patterns passed around in community discussions. Idiomatic code and real waste can share a bin, and bloat the rules do not name is missed. There is no follow-up in which people try to extend the repos and check that higher scores mean slower edits.
The human panel is a different task: at most 30 source-changing commits from each of 473 repositories. Agents also lose the conversation between checkpoints and keep only the directory, which is harsher than a session that retains chat. After a crash, later checkpoints score 0, while quality is computed only when code exists, so models that crash more often can look less degraded. Problems a frontier agent could solve in one shot were removed, so 14.8% does not transfer to small everyday edits. The prompt study covers three GPT models. Table 1 reports each model's best just-solve run. The prose and the table disagree on whether Opus 4.5 or Opus 4.6 has the 67.3% core rate.