Harness-Zero: Harness Distillation via Agent-as-Harness
Haoran Ye, Yuxing Lu, Haonan Dong, Zhaochen Su, Guojie Song
cs.AI, cs.CL, cs.NE
2026-09-22
Harness-Zero distills an evolved harness via a reviewer rewriting student replies. Qwen3.5-9B under mini Bash rises from 23.3% to 44.3%, beating 41.7% with the harness on.
A large share of agent performance sits in the external harness: tools, middleware, skills, memory. Methods such as Meta-Harness can evolve that scaffold, but the gains stay tied to the code you ship. The best harness changes with domain, instance, and base model. A general agent either shares a mediocre harness or routes among many specialized ones. Neither writes the discovery back into weights.
The evolved harness and the tiny production harness also differ in action space. Imitating source trajectories makes the student call tools that do not exist at deployment.
Three stages. Evolve a student-side harness h on training tasks, then adapt it into a private reference 𝒦 for a reviewer: blocking middleware becomes a review-time warning; tools become recipes for equivalent Bash under the target harness.
During collection the student always runs on a fixed mini-SWE-agent with one Bash tool. Each turn it proposes a reply. The harnessing agent, reading 𝒦, either passes it or makes the smallest coherent rewrite that is valid in the target action space. The review is hidden from the student. The reviewer cannot see hidden answers or the sandbox. The accepted reply executes through the target harness, and the observation enters the student trajectory.
LoRA SFT is applied to accepted replies, with reviewer-voice spans masked. At deployment, h, 𝒦, and the reviewer are gone. Only the distilled model and the mini harness remain.
Three domains: SpreadsheetBench Verified, AppWorld, and USPTO retrosynthesis. In the training-free comparison the same frontier model plays both student and reviewer. Agent-as-harness with evolved 𝒦 averages 81.1% across six settings, versus 78.1% for mounting the evolved harness as code and 68.6% for the mini harness alone. An empty 𝒦 scores 69.2%, so the gain is the evolved rules, not an extra onlooker.
The distillation student is Qwen3.5-9B, the harnessing agent GPT-5.6 Sol.
| Setting | Sheets | AppWorld | USPTO | Macro |
| Base + mini harness | 31.0 | 26.8 | 12.0 | 23.3 |
| Base + evolved harness | 39.0 | 48.2 | 38.0 | 41.7 |
| Harness-Zero (mini only) | 44.0 | 58.9 | 30.0 | 44.3 |
On sheets and AppWorld the distilled model under the mini harness already beats the base model that still wears h. USPTO rises from 12.0 to 30.0 but still trails 38.0 with h attached. Direct teacher rollouts, student rollouts under h, empty-𝒦 review, and even giving the reviewer the oracle answer yield 3%–15% test pass@1; Harness-Zero reaches 30%. Across 28 harness-exclusive behaviors, average recovery is 82.3%.
Harness engineering is now a primary lever for agents. This paper shows how to write the lever's output into weights: specialize the scaffold as a reviewer at train time, keep one mini interface at serve time. Teams that run many domain agents need not route a zoo of harnesses forever. Procedural habits (inspect before edit, scan after save) distill cleanly. Deep domain priors and executable validators do not, and USPTO is the counterexample.
At inference, agent-as-harness can also edge out a frozen code harness, if the reviewer and 𝒦 are strong enough.
A weak harnessing model can hurt. Collection adds a model call per turn; mean USPTO latency is about 2.4×, gone only after distillation. Mechanisms such as context management do not map onto student replies, so a mini harness remains. SFT throws away the rejected-versus-rewritten pair at the same state. All three domains run in Harbor containers; a raw desktop is untested. There is no RL distillation comparison.