Harness^2: Recursive Agent Harnessing for an Open World
Ruiyao Xu, Yanfei Chen, Zhongying CuiZhu, Bhavana Dalvi Mishra, Yifei Ming, Han Yu, Rujun Han, Chen-Yu Lee, Tomas Pfister
cs.AI, cs.CL, cs.LG
2026-10-10
Harness2 lets a frozen-model agent rewrite its own harness at test time via contrastive propose-probe-compose edits, gaining up to 11.5 points without ground-truth labels.
An agent is a model plus a harness: the system prompt, memory, skills, tool rules, and orchestration wrapped around frozen weights. The same model can score very differently under different harnesses. Recursive self-improvement (RSI, systems that make persistent changes to themselves so later improvements compound) has been applied to harness design, but the open world breaks it in two places. Tasks are unseen at design time and carry no ground-truth verifier, so nothing reliably says whether an online harness edit helped. And a harness tuned for one task transfers badly: inheriting it wholesale adds noise as the task distribution shifts. The paper pushes the question one level up: can RSI improve both the current harness and the way future harnesses get improved?
Harness2 freezes model weights and decomposes the harness into eight file-level components (system prompt, guardrails, memory, tool guidance, skills, scripts, subagents, a middleware plugin); every edit is a file diff. The framework plugs into stock OpenCode or Codex.
Task-level recursion is one contrastive step:
A budget k buys width (k candidates probed in parallel, k independent composition sessions) or depth (k chained steps). After the task, harnessing-level recursion reviews the full improvement record, extracts lessons backed by execution contrasts, and stores the adopted harness plus an EXPERIENCE.md in an improvement library. The next task restarts from the common base harness rather than inheriting the tailored one; the library only guides new proposals and composition, a defense against distribution shift. A hard guard keeps scores, rubrics, and grader verdicts out of the editor's working directory, so benchmark labels are reserved for final evaluation. The default editor is Claude Opus 4.8.
Three professional benchmarks, four configurations (GLM-5 and Gemini 3.5 Flash, each under OpenCode and Codex), means over three runs with shuffled task order:
| Config | Metric | Base | Harness2 (one step) | Gain |
| GLM-5 + OpenCode | WorkBuddy-Bench Avg | 64.2 | 75.7 | +11.5 |
| GLM-5 + OpenCode | LAB Partial | 59.6 | 71.4 | +11.8pp |
| Gemini 3.5 Flash + OpenCode | JobBench | 31.5 | 36.0 | +4.5 |
| Gemini 3.5 Flash + OpenCode | WorkBuddy-Bench Avg | 72.4 | 77.5 | +5.1 |
Parallel recursion at k=5 adds another 2.9-3.9 points on WorkBuddy-Bench (3.0-4.2 on JobBench, 2.6-6.7 points on LAB Partial); on WB-code and WB-office, both recursive forms at k=2 already beat every baseline at k=5. The baselines struggle: ReasoningBank drops as much as 6.6 points below base on coding tasks, Harness Scaling loses to base in 7 of 12 pairs, and Meta-Harness spends more compute than Harness2 at k=5 for lower scores, with Gemini+Codex on LAB as the one exception.
Ablations (Gemini + OpenCode, k=1): removing the improvement library costs 3.1 points on WorkBuddy-Bench and 1.8 on JobBench; removing the invariant/domain editor scoping costs 2.9 and 3.7; removing composition costs 1.6 and 1.9. Transfer runs answer two obvious objections. With the same model as executor and editor (k=2), all five models improve on every benchmark, GLM-5 by 13.7 points on LAB, so a stronger editor is not required. A harness refined for Gemini, moved to GLM-5 unchanged, gains 17.3 points on Code (58.5 to 75.8). A library accumulated on JobBench improves one-step refinement on WorkBuddy-Bench by 0.5-2.9 points. In the case study, a JobBench task whose base run failed procedurally (figures never reconciled across files, outputs never re-opened) gains a verification guardrail and a reconciliation skill, and goes from 43.1 to 77.8.
The practical takeaway for agent builders: when the budget is tight, tuning the harness can beat upgrading the model. GLM-5 with a refined harness (75.7) outscores Gemini 3.5 Flash in its stock harness (72.4). On WorkBuddy-Bench the gains do not buy longer executions; #Steps actually drops (25.4 to 21.9 on average for Gemini + OpenCode), and the k=5 version starting from OpenCode beats the strongest fixed production harnesses (Claude Code, Codex, CodeBuddy) by 4.6-5.8 points per domain. Code is open-sourced under google-research. This is solid incremental engineering rather than a new learning paradigm, but every design choice is backed by an ablation or a transfer study.
The paper dissects nine regressions itself, in three groups: the selector sometimes picks a weaker candidate (a 55 over an already-executed 95); a wrong premise propagates across candidates (one bad rule about dictionary keys dragged all four candidates down); judge crashes and bimodal criteria muddy attribution. The authors also concede the core constraint: behavioral contrast cannot verify correctness, and execution stochasticity plus shared model biases limit the evidence, an inherent weakness of the label-free setting. Beyond the self-reported ones: every main table uses Claude Opus 4.8 as editor, a stronger model than the executors, with same-model runs only at k=2; the k=5 budget is expensive and no cost figure is given; some small gains sit near run-to-run noise (GLM-5 + Codex gains 1.2 on JobBench against an SD around 2.3); libraries are per-domain and tasks run sequentially within a domain, so cross-domain online generalization rests on one offline experiment; and the leave-one-out reuse analysis is retrospective, with repeated execution possibly contributing to the difference.