Harness Handbook: Making Evolving Agent Harnesses Readable,Navigable, and Editable
Ruhan Wang, Yucheng Shi, Zongxia Li, Zhongzhi Li, Yue Yu, Junyao Yang, Kishan Panaganti, Haitao Mi, Dongruo Zhou, Leoweiliang
cs.AI, cs.SE
2026-07-15
An auto-built 'Harness Handbook' turns behavior localization in agent harnesses into a guided step, lifting edit-plan win rates to 45.6% on Terminus-2 while cutting tokens.
A modern agent's capability comes from more than its foundation model. The harness, the code that assembles prompts, manages state, invokes tools, and coordinates execution, does a lot of the work. As models, APIs, and requirements shift, the harness has to shift with them. Before a change, a developer or coding agent has to find every code location that implements the target behavior, and that is hard. Production harnesses span hundreds of functions across many files, a single behavior often lives in several non-adjacent spots wired together through shared state, and modification requests describe what the system should do while the repo is organized by file and module. Code search and long context help you look, but the behavior-to-code mapping is still recovered by hand. The paper calls this behavior localization and treats it as the central bottleneck in harness evolution.
The Harness Handbook is a behavior-centric representation synthesized automatically from a codebase. It is a three-level document tree: L1 is a system overview, L2 breaks components down by execution stage, and L3 links each unit back to source locations, plus a state-register view that records state shared across stages. Construction runs in three phases: phase one is deterministic static analysis that extracts functions, boundaries, and call edges with no model calls; phase two uses an LLM to organize source units into execution stages; phase three synthesizes the hierarchy. Every L3 locator is validated against the current repository, and anything that no longer resolves is frozen and excluded.
On top of it sits BGPD (Behavior-Guided Progressive Disclosure): start from stages, follow shared state to pull in coupled stages, select relevant entries, expand candidates along the call graph, then open the current repo and verify each one, keeping only locations that match real source. After edits, an automatic resync keeps the handbook aligned with the code. There are two leaf granularities: function-level when a trustworthy stage skeleton exists (Terminus-2), and file-level when the repo is large or no skeleton is available (Codex).
Two open-source harnesses: Terminus-2 (Python, 103 functions) in function-as-leaf mode, and Codex (a Rust monorepo, 2,267 files, 34,363 functions) in file-as-leaf mode, with 30 modification requests each across Query, Cross-file, and Search-Hostile types. The planner is DeepSeek-V4-Pro, judged by three independent models (GPT-5.5, Opus 4.8, DeepSeek-V4-Pro).
| Setting | Baseline win rate | Handbook win rate |
| Codex | 28.3% | 38.3% |
| Terminus-2 | 26.7% | 45.6% |
Overall win rate rises by 10.0 and 18.9 points respectively, while token cost drops 12.7% on Codex and 8.6% on Terminus-2. Against independent reference plans, localization F1 climbs across the board: on Terminus-2 versus an Opus 4.8 reference, symbol-level F1 goes from 64.8 to 77.1 and symbol-level wrong-location rate from 24.1% to 13.8%; versus a GPT-5.5 reference, symbol-level F1 reaches 89.3%. The largest gains show up for changes with scattered sites, cold paths, and cross-module interactions. The takeaway: a weaker planner with the handbook matches the localization quality of stronger models.
Anyone building agent harnesses or coding agents benefits directly. Harnesses get hard to change precisely because behavior and code do not line up; precomputing that mapping and keeping it synced amounts to a behavioral memory for the harness. The handbook does not do execution, only localization and planning, which happens to be the most painful step.
Building the handbook still needs LLM calls for organization, so it is not fully deterministic. Function-as-leaf mode depends on a reliable execution-stage skeleton that not every repo has. The paper evaluates only localization and planning, handing execution to a separate agent, with no end-to-end check that the modified code is actually correct. Both tested harnesses are specific projects, so generality across more repos remains open.