How Do Coding-Agent Harnesses Self-Evolve? An Empirical Study of Retrospective Adaptation
Christoph Treude, Sebastian Baltes
cs.SE, cs.AI
2026-10-26
Claude Code ran 30 SWE-bench issues, /retro proposed 125 harness changes; of 12 evaluated individually, only 2 eliminated the targeted repeated work, and some added work on unrelated issues.
Coding agents such as Claude Code run inside a harness: the software, configuration, and instructions around the agent, including context files like CLAUDE.md that are read at the start of every session. When a session learns how a repository's tests run, or which tests already fail, it can write that knowledge into the harness for later sessions. Retaining such experience without retraining the model is what the paper calls harness self-evolution.
Whether a retained change helps is a separate question. It may hold only for that one task, may never enter a later session's context, or may be read without changing what the agent does. The /retro skill, from Matt Pocock's public skills repository, automates the generation half of the loop: after a session ends, it has the agent review that session and produce numbered harness-change proposals for human approval. The verification half, one change at a time on later tasks, did not exist. This study supplies it.
Three stages.
Evaluating one adaptation at a time is the design decision that matters. Earlier work stacked many changes together, which makes attribution impossible.
Of the 125 proposals, 101 were experience-driven, 19 prior-driven, 5 weak. Executable checks led the mechanisms with 38, ahead of navigation pointers (24), instructions (21), information-access changes (20), and tool or configuration changes (12); 10 proposals changed no harness at all. Environment trouble was the backdrop: 19 of the 20 second-sample traces contained a failed or repeated shell command or multiple test runners, and sessions in 15 of 30 tasks were offered known-failures files.
The coding already shows cracks. Only 9 of the 38 executable checks could report failure in the task environment as written; 28 of the remaining 29 needed missing tools or were commit hooks that never fired because the sessions made no commits. And 18 of the 43 lint-related proposals were prior-driven, recommending lint regardless of what had happened, because the skill's instructions define a repository without a pre-commit hook or CI as 'a finding'. Retrospective output mixes two things: facts learned from the session, and generic practice the skill's author shipped in advance.
Seven of the 13 transfer events were informative:
| Adaptation | Original harness | Adapted |
| SymPy CLAUDE.md line naming bin/test | 5/5 | 0/5 |
| Astropy known-failures file (3 tests) | 3/5 | 0/5 |
| Astropy cache-repair plugin + script | 5/5 | 2/5 |
| Sphinx known-failures files (2 adaptations) | 5/5 | 5/5 |
| SymPy two-package known-failures file + script | 4/5 | 4/4 |
| pytest known-failures file (3 tests) | 5/5 | 5/5 |
The two adaptations that eliminated the targeted action were exactly the two that covered the instances later sessions hit: a test command that stays valid across issues, and a failure list matching the failures encountered. Uncovered lists were read in every adapted session, and the agent still reran baseline comparisons (one adapted session ended on an authentication error, hence 4/4). Task success never differed by more than one session between harnesses; median time fell by 5 to 28 seconds in the four informative events without uncovered failures or bundled instructions.
Side effects were real. One Sphinx file told the agent to treat any unlisted failure as its own doing; on an unrelated control issue, 4 of 5 adapted sessions ran baseline comparisons against 0 of 5 under the original harness. The effective SymPy line came bundled with an instruction to run CI lint before finishing; every adapted session obeyed, and median elapsed time rose from 88 to 156 seconds, while the Sphinx control task went from 82 to 245 seconds. The author-written pre-commit configuration fared worst: present in all five adapted sessions, read by none.
For people running Claude Code or Codex daily, this stress-tests a fashionable practice: letting an agent retrospect and grow its own CLAUDE.md, hooks, and scripts. The answer is conditionally yes. Repository-level facts such as how to run the tests transfer; one-off failure lists mostly do not, because later sessions hit different failures. If you use a skill like /retro, count how much of its output is prior-driven generic practice rather than something learned from your sessions.
For teams building self-evolving agent systems, the evaluation method may be more useful than the findings: one change at a time, record whether the problem arose and whether the change covered it as separate facts, and run controls on unrelated tasks to catch added work. The conclusion extends to any memory or skill-retention mechanism: retention needs verification and a way to retire changes, or ineffective ones sit in the harness indefinitely and sometimes add work.
The authors' own list: evaluation issues were selected to reproduce the targeted problem, which overestimates how often adaptations help on unselected tasks; changes were implemented inside the original session's container, so files could carry information beyond the proposal text; the first author's labeling was not fully blind, since traces reveal whether transferred files were read; and five sessions per harness across seven informative events describe cases rather than establish an explanation.
Two boundaries deserve to be stated bluntly. The sample skews toward environment problems, because the second batch of issues was chosen for exactly that, so navigation and coding-convention adaptations were never evaluated; harness self-evolution was tested on one corner of it. The qualitative coding came from two LLMs, and the paper itself notes the agreement reflects consistency across two inputs, not human inter-rater reliability. The study covers one agent (Claude Code) and a single commit of /retro.