HarnessCompass: Guiding Automatic Harness Evolution toward Generalizable and Effective Agent Harnesses
Luan Zhang, Ruochen Zhou, Dandan Song, Zhengyu Chen, Yuhang Tian, Jun Yang, Huipeng Ma, Chenhao Li, Guangyuan Feng, Xudong Li, Yizhou Jin, Yan Xu
cs.LG, cs.CL
2026-08-03
Three rules discipline auto-evolution of an agent's harness. GPT-5.4's Pass@1 on SWE-bench Verified climbs 54% to 66% in 5 turns, and the evolved harness transfers to unseen tasks and Claude-Sonnet-4.6.
Whether a coding agent can fix a bug depends on two things: the base model, and the harness wrapped around it. The harness is the software layer between model and environment, covering the system prompt, tools, middleware, memory, and verification logic that decide how the model sees state, calls tools, and recovers from failure. Prior work has shown that swapping the harness alone can match the gain from upgrading the base model.
The catch is that the best harness is model-specific and must be re-tuned whenever the model changes, and this tuning has always been manual: read trajectories, diagnose failures, revise the harness by hand. As models iterate faster than humans can keep up, automatic harness evolution emerged. A meta-agent analyzes task trajectories, proposes harness edits, evaluates, and iterates. Methods like AHE report large gains on public benchmarks, but the authors audit where those gains come from and find three flaws.
First, they overfit the search tasks, because the meta-agent is free to revise against a fixed task set and bakes in shortcuts that only work there. Second, they rely solely on trajectory-outcome signals: the meta-agent sees that a task failed and where, but not why the harness was hard to use, so it often blames the model's own reasoning failures on the tools. Third, they edit all components jointly, so prompt, tool, middleware, and memory edits interfere, and gains that should compound instead cap out.
HarnessCompass prescribes one remedy per flaw, combined into a disciplined loop.
The first is a generalization gate (constrained evolution). Every candidate edit must pass a global gate that rejects anything naming a specific task instance, test function, private symbol, or task-specific token branch, admitting only transferable criteria with an applicability condition the agent can evaluate on an unseen task. The gate also fixes placement: capability edits that add executable functionality must land in middleware, tools, or sub-agents, while guidance edits that only advise behavior must land in the system prompt or memory, never as executable code. To guarantee every gain is earned by the loop, the authors start from a deliberately minimal harness: one shell command and a short prompt, no middleware, skills, or sub-agents.
The second is proactive feedback. After a task fails, the same code agent that ran the trajectory reports where the harness hindered it, in two queries: a blind report (verdict withheld, to avoid hindsight rationalization) and a hindsight report (verdict revealed, attributing the failure to harness, agent reasoning, task ambiguity, or environment). Each report is then checked against the trajectory and only kept if the trace supports it, discarding spurious complaints and misattributions. This gives the meta-agent a "why" signal that raw traces cannot.
The third is component-wise optimization. Each round evolves two variants in parallel, one touching only structural components (middleware, tools, sub-agents) and one only guidance components (system prompt, skills, tool descriptions, memory), and picks the higher Pass@1 as winner. The loser is not discarded: an R3 merge keeps independently beneficial, non-conflicting loser edits (Revision), applies them onto the winner with winner precedence in conflicts (Recombination), and drops redundant overlaps (Refinement). The tracks do not interfere, yet complementary gains survive.
The main experiment runs on SWE-bench Verified, 500 real GitHub issues, with 50 as the evolution set and 450 held out unseen throughout, using GPT-5.4 in non-thinking mode.
| Method | Evolution set (50) | Held-out (450) | Total (500) | Turns |
| Minimal seed H0 | 54.0% | 51.6% | 51.8% | 0 |
| AHE | 63.0% | 54.7% | 55.5% | 20 |
| HarnessCompass | 66.0% | 60.4% | 61.0% | 5 |
The gap is larger on held-out tasks (5.7 points) than on the evolution set (3.0), so the gains come from transferable design rather than search-set shortcuts. A cumulative ablation adds the principles one at a time: the gate alone lifts the seed from 54% to 62% in two turns and held-out from 51.6% to 58.4%; adding proactive feedback pushes the evolution set to 66% but drops held-out to 55.8% over 12 turns (it begins to overfit); adding R3 keeps 66% on the evolution set while restoring held-out to 60.4% in 5 turns.
Cross-model transfer is the harder test. The harness evolved on GPT-5.4 is frozen and run unchanged on Claude-Sonnet-4.6 (non-thinking mode): total Pass@1 rises from 70.0% to 73.8%, held-out from 70.2% to 73.6%. Broken down by repository, sphinx-doc climbs 35.2% to 58.0%, pytest-dev 52.6% to 68.4%, django 59.3% to 70.1%, while an already-strong scikit-learn (73.4%) barely moves; on Claude, astropy goes 45.5% to 63.6%. Gains concentrate on the repositories with multi-step edit-and-verify loops that depend most on the harness.
For teams deploying coding agents, the harness is a far cheaper lever than training a new model. HarnessCompass offers not another benchmark number but a reusable discipline: evolve on a small search set and get a harness that transfers to unseen tasks and other models. It turns scaffold tuning from a craft into a reproducible engineering step. Honest reading: the gains are real but not revolutionary, lifting a base already at 54%; the bigger value is efficiency (5 turns versus 20) and transferability.
The evaluation surface is narrow. It covers only SWE-bench Verified, Python-repository English software-engineering tasks across twelve repos; whether the discipline holds for non-coding agents (web, terminal, tool orchestration) is untested. Cross-model transfer is tried on only two frontier models (GPT-5.4, Claude-Sonnet-4.6); behavior on smaller or open models is unknown. Only two component tracks are split; finer decomposition could help or hurt. Held-out still trails the evolution set (60.4% versus 66%), so some search-set affinity remains. Proactive feedback roughly doubles the model calls per failed task, a real cost. All evaluation runs in non-thinking mode to isolate the harness, while real deployments often use thinking, where the harness contribution may shrink.