Agentic Harness Engineering: Observability-Driven Automatic Evolution of Coding-Agent Harnesses
Jiahang Lin, Shichun Liu, Chengjun Pan, Lizhi Lin, Shihan Dou, Zhiheng Xi, Xuanjing Huang, Hang Yan, Zhenhua Han, Tao Gui, Yu-Gang Jiang
cs.CL, cs.SE
2026-04-29
A Fudan loop auto-edits a coding-agent harness. Ten rounds lift Terminal-Bench 2 pass@1 from 69.7% to 77.0%, beating Codex; the frozen harness transfers across models.
Coding-agent scores on long-horizon repo repair and terminal workflows are not a property of the base model alone. The surrounding harness (system prompt, tools, middleware, skills, sub-agents, long-term memory) often sets the ceiling. When the base model changes, that layer usually has to be rebuilt. Today the rebuild is manual: people read traces, spot recurring failures, and hand-edit configs. Models move faster than that loop.
Automation is stuck in a narrower place. ACE distills a natural-language playbook the agent rereads in context. Training-Free GRPO reinforces successful tool sequences. Neither opens the scaffolding around the model. Jointly editing every component hits two walls: raw traces bury the usable signal under millions of tokens, and coupled frameworks make a one-file fix scatter side effects that nobody can attribute.
AHE treats the bottleneck as observability, not as a stronger evolution agent. Three pillars close a loop while the base model stays frozen and only the explicit harness is edited.
Component observability is NexAU. Seven editable types live as files in one workspace: system prompt, tool description, tool implementation, middleware, skill, sub-agent config, and long-term memory. Failure patterns map to a single component class; each logical edit is a git commit, so rollback is file-granular. The seed harness is deliberately thin: one shell tool, no middleware, no skills, no sub-agents. A seed already fitted to the benchmark would contaminate attribution.
Experience observability is Agent Debugger. Raw rollouts sit at roughly ten million tokens. The debugger treats traces as a navigable filesystem, writes a per-task root-cause report, then rolls those reports into a benchmark-level overview. The evolve agent reads the overview first and drills into raw traces only when a claim needs checking.
Decision observability is a change manifest. Every edit names the evidence, the inferred cause, the tasks it expects to fix, and the tasks it flags at risk. The next round intersects those sets with observed task-level deltas and reverts rejected edits at file granularity. The evolve agent may write only inside the harness workspace. Verifier, model, and reasoning budget are read-only, so a recorded gain cannot come from swapping the model or disabling the checker.
All three roles share GPT-5.4 at the high reasoning setting, which pins the gain on harness edits. One outer iteration is: rollout, clean, attribute the prior manifest and roll back, distill, edit, commit. Each task gets at least two rollouts so partial-pass items can support comparative diagnosis.
Evolution runs on all 89 Terminal-Bench 2 tasks (4 easy, 55 medium, 30 hard) with a 1-hour per-task timeout. Ten iterations take about 32 hours.
| Method | All | Easy | Medium | Hard |
| OpenCode | 47.2% | 75.0% | 52.7% | 33.3% |
| Terminus-2 | 62.9% | 75.0% | 74.5% | 40.0% |
| Codex | 71.9% | 75.0% | 80.0% | 56.7% |
| NexAU0 seed | 69.7% | 87.5% | 78.2% | 51.7% |
| ACE | 68.9% | 91.7% | 78.2% | 48.9% |
| TF-GRPO | 72.3% | 100.0% | 79.4% | 55.6% |
| AHE | 77.0% | 100.0% | 88.2% | 53.3% |
On Hard, AHE trails Codex. Dropping only AHE's long-term memory into the seed already reaches 63.3% on Hard, above Codex. The gain does not live in the prompt: swapping the system prompt alone falls to 67.4%. Memory, tools, and middleware alone reach 75.3%, 73.0%, and 71.9%. Those three positive single-component gains sum to +11.1 pp against full AHE's +7.3. Memory, middleware, and the prompt all push the same closure-style check, so stacking them burns turns inside the long-horizon budget. Because 55 Medium tasks dominate the aggregate, the loop returns part of the Hard memory effect.
Frozen, with no further evolution, the harness moves to SWE-bench-verified (500 tasks): 75.6% aggregate against the seed's 75.2%, at 12% fewer tokens (461k vs 526k). ACE and TF-GRPO fall below the seed and spend 11% to 29% more tokens. The Terminal-Bench playbook is cost without policy change on a different task surface. Cross-family bases all gain: deepseek-v4-flash 51.7% to 61.8% (+10.1), qwen-3.6-plus 56.2% to 62.5% (+6.3), gemini-3.1-flash-lite-preview 36.5% to 41.6% (+5.1). Inside the GPT-5.4 family, medium and xhigh each pick up only +2.3. Step budget and timeout were fitted to the high setting; xhigh pushes more trials past the timeout, which the metric counts as failure.
Self-attribution is lopsided. Fix precision 33.7% and recall 51.4% sit about 5x above random. Regression precision 11.8% and recall 11.1% sit only about 2x above random. The agent can name why an edit should help. It cannot name who the same edit is about to break.
For teams shipping coding agents, the usable lesson is operational: freeze the base model, expose tools, middleware, and memory as revertible files, and drive edits from distilled traces. Prompt-only self-evolution is the wrong layer. ACE and TF-GRPO barely beat the seed on Terminal-Bench 2 and regress on SWE-bench.
This is incremental engineering, not a new model. It fits teams that already have a reproducible eval and can split the harness into files. Cross-model gains are larger on bases further from saturation; weaker models lean harder on coordination that has been written into tools and memory. The SWE-bench headline is a 0.4 pp bump. What actually travels is the token cut and the lift on django and sphinx-doc, whose edit-and-verify loops match the structure AHE compressed.
The authors list three. Evolution is Terminal-Bench 2, transfer is SWE-bench-verified; broader languages, repo-scale deploys, and human-in-the-loop workflows are untested. Step budget and timeout are fitted to GPT-5.4 high, so within-family non-monotone gains mix harness portability with operating-point coupling. Governance bounds edits to a workspace; long-horizon cleanup and misuse prevention are incomplete. They call it a controlled research prototype.
Two more discounts follow from the tables. The panel is 89 tasks, Hard is 30, and Easy in the ablation is 4, so percentages jitter. The objective is Medium-heavy; memory-only beats full AHE by 10 pp on Hard. The loop is trading for Medium. Regression foresight is the missing piece, and the non-monotone steps on the evolution curve are the symptom. If the next paper adds one capability, it should be regression prediction, not another component.