Harness-R1: Learning to Edit Executable Runtime Harnesses from Agent Failure Trajectories
Shuai Shao, Kangning Zhang, Qingyao Li, Shijian Wang, Hao Wang, Wenxiang Jiao, Yuan Lu, Yi Guo, Weiwen Liu, Weinan Zhang
cs.AI
2026-08-03
A 9B 'harness engineer' reads a frozen agent's failures and writes executable runtime patches; outcome RL raises Qwen3.5-9B from 44.3% to 53.6% averaged over WebShop, ALFWorld, and DBBench.
LLM agents keep accumulating interaction trajectories after deployment, but their behavior stays fixed. Getting better has meant one of two things: update the model's weights, or edit the harness wrapped around it, the scaffolding code that assembles context, calls tools, validates actions, and recovers from errors. Updating weights is expensive; editing the harness looks cheap, but has never had a reliable automated method.
Existing approaches are either fixed rules (like Self-Refine, where the agent reflects on itself) or handing a strong model the editor job. The authors rule both out. A fixed Self-Refine rule lowers reward on all three benchmarks they test. And when GLM-5.2 or GPT-5.5 is asked to edit the harness, the patches come out syntactically valid and reasonable-looking, but because no one ever reruns the target agent, the editor cannot tell whether the patch actually raised success. The editor itself had never been trained on real outcomes.
Harness-R1 turns harness editing into a learned capability optimized on real success and failure. It leaves the target agent's weights frozen and trains a separate 9B "harness engineer" model.
Two stages. Cold start: a GPT-5.5 teacher writes candidate patches from batches of target-agent failures, each patch is actually installed and the frozen target is rerun, and only patches that execute, complete, and don't regress other tasks are kept. That yields 877 examples (WebShop 381, ALFWorld 248, DBBench 248) for supervised fine-tuning, giving the engineer a prior on what a legal patch looks like.
Then online RL with GRPO. For each failure batch, the engineer samples K=8 candidate patches, each is parsed, installed, and the frozen target is rerun on the whole batch; the reward is the change in batch-average reward before and after the patch, and candidates that fail to parse, do nothing, or don't complete get zero. The eight candidates are normalized group-relative into advantages, only the engineer's parameters update, and the target stays frozen. Reward is computed within the same batch with no memory across batches, and no per-trajectory polishing.
The patch itself is an executable overlay hooked into the agent's main loop at four lifecycle points: episode start (oninit) injects opening guidance and tool hints, pre-decision (makeprehint) adds state-conditioned hints, pre-action (onbeforeaction) blocks or rewrites a clearly bad action before it reaches the environment, and post-feedback (onpoststep) triggers recovery after the environment responds. This lets the engineer intervene on context, actions, and recovery without touching model weights.
Main table, target agent = Qwen3.5-9B:
| Method | ALFWorld | WebShop Succ. | DBBench Succ. | Avg. |
| Default harness | 40.6 | 31.2 | 61.0 | 44.3 |
| ReAct | 43.4 | 37.4 | 61.7 | 47.5 |
| Self-Refine | 39.0 | 29.0 | 57.3 | 41.8 |
| GLM-5.2 as editor | 45.0 | 36.0 | 65.3 | 48.8 |
| Supervised-only engineer | 39.4 | 38.6 | 61.3 | 46.4 |
| Harness-R1 | 53.2 | 42.2 | 65.3 | 53.6 |
| Agent SFT + Harness-R1 | 84.0 | 43.0 | 65.7 | 64.2 |
Key readings. Harness-R1 raises the frozen target from 44.3% to 53.6% average, up on all three benchmarks. It beats the supervised-only engineer by 7.1 points, so the gain is from online RL, not just imitating the teacher. It also beats GLM-5.2 used as a frontier editor (48.8%). And after the target agent itself is fine-tuned to 59.2%, a retrained target-specific engineer adds another 5.0 points to reach 64.2%.
Generalization is the stronger part. Across 20 unseen target configurations (different models, sizes), the benchmark-averaged gain is 7.06 points, every target-level average is positive, and 56 of 63 target-benchmark combinations improve, with the three regressions all under 2.0 points. On held-out tasks, where the engineer sees the same 10 failures and writes one patch applied to 1,270 other tasks, Harness-R1 still gains 8.9±1.5 points, while Qwen3.5-397B and DeepSeek-V4-Pro as editors come in at −4.3±2.5 and −0.4±3.6. Turning a handful of failures into a broadly useful edit is not something scale alone buys.
The lifecycle ablation removes each of the four positions. Pre-action mediation costs 3.9 points when removed, post-feedback recovery 3.3, while episode-start and pre-decision changes cost only 0.9 and 0.6. The dominant position varies by environment: pre-action matters most on WebShop, post-feedback recovery most on ALFWorld.
For anyone shipping agents, this route is cheaper and more realistic than fine-tuning the model. The target can stay frozen; what changes is the scaffolding around it, driven by a 9B engineer that beats much larger frontier editors precisely because it is trained on outcomes rather than scale. The same training recipe transfers across model families, so equipping a new agent is bounded work.
It is not a silver bullet, though. The engineer is fit to one specific target, mined from that agent's own failures; switching agents means re-mining and retraining.
The authors flag three. First, they study only a single adaptation step from a vanilla target to a fine-tuned one; true multi-round co-evolution of engineer and target is left to future work. Second, reward comes only from same-batch outcomes, so the signal is tied to the tasks used to mine failures, with no explicit term against regressing on unseen tasks. Third, inference cost is ignored, so some patches may trade runtime for accuracy.
Two gaps the paper does not close. All three benchmarks are closed and deterministic (shopping, simulated household, SQL), and the four hooks assume a clean "rerun the same batch" reward signal; whether that premise holds for open-ended web, long-horizon, high-uncertainty settings is untested. The 877 cold-start examples depend on GPT-5.5 filtering, so the teacher's biases flow into the engineer.