Harness-of-Harness: Multi-Day Autonomous Software Development with Continual Improvement
Haoyang Yan, Min-le Su, Hangfan Zhang, Zhanhao Li, Chen Zhang, Shao Zhang, Yang Chen, Lei Bai, Shuyue Hu
cs.AI
2026-09-02
HoH wraps existing coding harnesses in plan-code-test loops: +52.25% mean relative gain at 3 iterations; Codex GameCraft 49.58→71.52.
Most coding agents still sit in a human loop: people split tasks, watch intermediate steps, and step in on failure. This paper aims at a harder setting. Given only high-level requirements, agents start from an empty repo and finish a runnable system, with no further human guidance.
Long trajectories make the failure modes worse. Earlier requirements drop out of view, local patches violate constraints elsewhere, failed attempts pile up, new test evidence invalidates old assumptions, inspect-and-repair cycles spin, and the agent declares done while pieces are missing. Stretching a single session does not by itself keep progress coherent across days.
Harness-of-Harness (HoH) leaves existing coding harnesses unmodified and wraps them in a planning–coding–testing loop. The same model and harness are invoked three times per iteration, under different roles:
Cross-loop state is split. Artifact state \(At\) holds code, assets, and config. Evidence state \(Et\) holds verified behavior, gaps, and failures. The next planner reads \(Et\); the next developer continues from \(At\). Context uses progressive disclosure: full artifacts live on disk, a short index is shown first, details are retrieved on demand. There is no extra memory module. The protocol constrains verifiable outputs, not the agent's internal workflow.
Benchmark runs disable extra tools and version control, measuring the protocol alone. The multi-day case then turns on Godot MCP, asset tools, skills, and GitHub.
Three harness–model pairs: Codex with GPT-5.5 (high), OpenCode with DeepSeek-V4-Pro, and Pi with MiniMax-M3. The baseline is a single Vanilla pass of the same pair. After three iterations the mean relative gain is 52.25%, and the maximum is 82.86% (Pi on FrontierSWE Dominance, 35% to 64%).
| Setup | GameCraft Overall | FrontierSWE Dominance | ProgramBench Pass Rate |
| Codex Vanilla → HoH@3 | 49.58 → 71.52 | 44% → 71% | 60.41 → 66.50 |
| OpenCode Vanilla → HoH@3 | 26.90 → 48.98 | 25% → 44% | 45.27 → 57.56 |
| Pi Vanilla → HoH@3 | 42.16 → 58.78 | 35% → 64% | 35.83 → 52.68 |
Mean FrontierSWE rewards move from 0.31, 0.23, 0.26 to 0.54, 0.31, 0.55. All four GameCraft rubric parts (mechanics, content, functional visuals, presentation) rise under every pair. Codex run to 10 loops, scored in a pool of Vanilla plus HoH@1–10, reaches 72.67% Dominance at HoH@10 (76.00% at HoH@9) against Vanilla at 27.33%.
A matched-pass control rules out "just run it more times." On GameCraft with Codex, three-pass Vanilla Continuation scores 58.24 at 6.33M tokens. Two-pass HoH scores 64.84 at 5.67M tokens, already ahead of the three-pass continuation. Three-pass HoH reaches 71.52 at 8.41M tokens. Ablations on Codex with \(T=3\): freeze the first plan −8.13, replan without evidence −6.28, rebuild from an empty workspace each loop −7.85 and tokens rise from 8.41M to 11.12M.
The multi-day case, Fusepoint, starts from an empty workspace plus a PRD. Codex with GPT-5.6-Sol produces a playable narrative FPS in 70 loops. Humans only restored network and API access; they did not plan, implement, debug, or accept. Of 81 issues, 65 closed, 16 remained open, and 17 reopened after a later regression.
The lift comes from outer-loop state, not a stronger base model. Three Vanilla setups at very different levels all move, which means the protocol can sit on existing harnesses. For anyone building long-horizon coding agents, four pieces transfer: increments must be verifiable, implementation and acceptance stay apart, artifact and evidence both carry forward, and the plan is rewritten from evidence each round. Extra tokens buy an end to spinning in place under naive continuation.
This is still an incremental systems result. HoH adds no new model and no new single-pass coding skill. It organizes an existing harness into an iterable project.
GameCraft uses 45 of 140 tasks, FrontierSWE 15 of 17; the samples are small. Benchmark HoH turns off the tools and version control used in the case study, so the two experiments are not the same system. Fusepoint is a single case whose success test is "a human can play," with no 70-loop alternative protocol as control. QA evidence is withheld from official benchmark scorers to avoid leakage, which also means the loop never sees the final score. On ProgramBench, Pi peaks at HoH@2 (53.57) and drops to 52.68 at HoH@3, so gains are not monotone. Token counts include caches and should not be compared across vendors as cost. Humans do not write code, but they still repair outages; fully unattended operation is not shown.