Evo-Harness: Context-to-Harness Skill Compilation for Self-Evolving Agents
Tianxin Wei, Zhan Shi, Minhua Lin, Bing He, Zewen Liu, Yisi Sang, Yuanchen Bei, Xuying Ning, Jiaru Zou, Ting-Wei Li, Xiao Lin, Yanjun Zhao, Chi Wang, Benoit Dumoulin, Dakuo Wang, Jingrui He, Hanqing Lu
cs.AI, cs.CL
2026-08-15
A frozen solver compiles each failed run into general and topic skills. Evo-Harness beats XSkill on five benchmarks; TerminalBench-2 rises from 62.9% to 73.0%.
When an agent fails, most systems dump the trajectory into a memory store and retrieve it later. Real deployments see a stream of novel tasks. Each run is a one-shot chance to learn, and the context is noisy: partial errors, incidental tool traces, one-off paths. Retrieving past cases often injects that noise into the next prompt.
Existing reflection, memory, and skill methods rarely isolate what actually drives improvement on hard realistic tasks. Many of them mine a pool of trajectories offline. That is not the same as learning immediately after each run.
The setting is online harness learning. Solver weights stay frozen. Only an external harness is updated. The harness is a structured set of reusable guidance entries, not a raw trajectory log.
Work proceeds in batches. For each task the system selects at most b harness entries, injects them, and lets the frozen solver run. The execution yields a trajectory, an outcome, and environment feedback. Reflection fires only on failure or negative feedback and writes a candidate memory: lesson, trigger, evidence, scopehint. Successful runs do not update the harness, so lucky task-specific details stay out.
After a batch, an evolver compares those memories with the current harness and applies Add, Merge, Revise, or Skip. Updates sit at two levels. General skills capture cross-task operational patterns. Topic skills keep local procedures for a task type. The levels are not pre-assigned labels; they emerge as the evolver filters and merges.
The stack is modular on purpose: solver, evolver, feedback source, and initial harness can be swapped, so each factor can be measured. The default solver and evolver are Claude Opus 4.6.
Evo-Harness is best on all five benchmarks, including against the strongest baseline XSkill and a No-Evolve control:
| Method | CL-Bench | TerminalBench-2 | SWE-bench Lite | τ-bench | WebArena-Infinity |
| No Evolve | 29.54 | 62.92 | 63.67 | 72.73 | 72.50 |
| XSkill | 31.44 | 66.29 | 64.67 | 73.94 | 73.75 |
| Evo-Harness | 34.02 | 73.03 | 67.00 | 76.97 | 76.25 |
The largest jump is TerminalBench-2, where agents inspect files, run commands, read errors, and recover. Several baselines fall below No Evolve on some suites, so stuffing in experience can hurt.
On CL-Bench, Procedural Task Execution is the reliable win: Opus 4.6 and 4.5 each gain +9.8. Empirical Discovery and Simulation is unstable; Opus 4.6 drops 2.5 points. Stronger solvers absorb the harness better: Opus models gain +3.7 to +4.5, Kimi-K2.5 +1.1, GPT-OSS +0.8.
Ablations split by task type. On CL-Bench, Topic Only reaches 33.70 against the full 34.02, while General Only falls to 30.28. On SWE-bench Lite the reverse holds: General Only 66.67, close to 67.00. Heterogeneous reasoning wants local procedures; repo-level debugging wants shared inspection and verification habits.
Feedback has to be grounded. Self-judged success drops CL-Bench from 29.54 to 27.96 and SWE-bench Lite from 63.67 to 61.67. Pass/fail from the environment is enough for SWE at 67.33. Richer diagnostic feedback lifts CL-Bench to 34.02 while SWE stays at 67.00. Diagnostics help reasoning; overly specific traces can pin skills to one failure.
Skills transfer across splits and models: on SWE, No Evolve is 68.8, a Sonnet-evolved train-split harness scored by Opus 4.7 reaches 73.4, and online updates reach 75.0. When Sonnet is the solver, evolution lands at 55.3 and 55.7, both below the 58.0 No-Evolve score. Skills help a solver that can follow them. They do not help one that cannot.
For anyone shipping agents, this is a no-finetune path: compile a failure into the harness and reuse it on the next similar task. The conditions are concrete. The task needs reusable procedure, the environment needs real feedback, and the solver needs to be strong enough to read the guidance.
It also turns self-evolution from a single score into something you can take apart: evolver design, feedback grain, and cross-model transfer can be measured one at a time.
The study is single-agent text, tools, web, CLI, and software engineering. No embodied or multi-agent settings. The harness is natural-language guidance, not executable code skills. The main solver is Opus 4.6; open-weight gains are small, so adding a skill store is not a free lunch. The EDS drop on CL-Bench is a warning that exploratory tasks can be steered wrong by over-specific skills. Online updates run on the test stream itself, which is not a clean train-test split.