Evo-Bench: Can Language Models Improve Agent Harness?
Lisheng Huang, Chen Yang, Hao Zhou, Huatong Song, Zongchao Chen, Ran Le, Yang Song, Wayne Xin Zhao, Tao Zhang
cs.CL
2026-08-10
With the policy model fixed, nine frontier models iteratively rewrite an agent harness; GPT-5.6 gains 16.6 points, beats human code on general tasks, yet trails 46.3 to 47.5 overall.
An agent's ability on long-horizon tasks comes from more than the base model. The harness (how it assembles context, calls tools, plans, and recovers) accounts for a large share of the gap between products like Claude Code and Codex. As models strengthen, a natural question follows: can a model improve its own harness instead of waiting on human engineers?
Prior work tuned prompts, optimized workflows, or let agents self-evolve, but none could measure this cleanly. Task scores reflect both harness quality and base-model strength, and the two confound. Models also overfit the validation split. Evo-Bench targets exactly these three problems: select tasks that respond to harness changes, keep validation and evaluation splits aligned in that response, and force multi-round iterative refinement.
The design fixes the policy and swaps the evolver. A fixed policy model (DeepSeek-V4-Flash in the main runs) executes tasks through a minimal seed harness H0: a CodeAct loop with only shell execution and a finish tool, no planning, memory, or verification. The model under test plays harness engineer, diagnosing failures, forming a hypothesis, editing code, and evaluating on validation, repeating for up to 20 iterations, 48 hours, and 1000 steps. The final harness is frozen and scored on a held-out suite it never saw.
The hard part is task construction, handled by a harness-guided two-stage framework. First, four frontier models evolve on auxiliary tasks, producing 73 harness variants; dedup leaves 12 representative auxiliary harnesses. These 12 then run 2329 candidate tasks. For each task, the benchmark computes harness sensitivity, the Pearson correlation between the task's score and overall harness quality. Tasks with non-positive sensitivity are dropped because they cannot distinguish good harnesses from bad. The rest are stratified by difficulty, ranked by sensitivity, and split into aligned validation (160) and evaluation (448) suites.
This alignment is what lets validation optimization predict held-out performance instead of overfitting.
Leaderboard across nine models (seven frontier, two open-weight):
| Model | Overall | vs CodeAct seed |
| GPT-5.6 Sol | 46.3 | +16.6 |
| Claude Opus 4.8 | 45.8 | +16.1 |
| GLM-5.2 | 43.5 | +13.8 |
| CodeAct (seed) | 29.7 | baseline |
| Human-engineered | 47.5 | n/a |
Top models lift a near-bare harness by more than 16 points, within reach of human engineers but not past them. Gains are uneven across domains. Search jumps almost universally (Claude +34.8, nearly matching humans) because web-navigation logic is easy to synthesize. Office is stuck: most models move marginally or regress, since these tasks need highly specific processing workflows. On General tasks, GPT-5.6 Sol and Qwen3.7-Max surpass the human-engineered baseline (59.4 vs 56.3), the first clear case of autonomously evolved reasoning structures beating manual design.
Cost spreads are extreme. GPT-5.6 Sol costs over $500 per run; GLM-5.2 and Qwen3.7-Max land near the top for under $40; DeepSeek-V4-Pro synthesizes usable improvements for less than a dollar. The Pareto frontier is a steep logarithmic curve with diminishing returns.
The clearest pattern is early saturation: models find good structure in the first few rounds, then introduce harmful changes. Claude Opus 4.8 and GLM-5.2 post the highest Anytime Validation scores, precisely because they peak early and decline.
This turns the idea of models improving their own harness from an anecdote into a measurable, comparable capability. For practitioners: harness optimization has real headroom and models can mine it automatically, especially for Search and General tasks; Office-style procedural tasks still need humans for now. The cost curve shows you do not need the most expensive model; GLM-5.2-class models offer the best value.
Cross-policy transfer experiments (swapping in Qwen and GLM as the policy model) show the evolved harnesses are transferable reasoning structures, not overfits to one model's quirks. That is the key evidence for reuse.
The authors are blunt. Evolvers do aggregate hill-climbing, not causal diagnosis: they watch the total score move without isolating the root cause of a failure. They resolve cross-domain interference with naive routing instead of robust shared mechanisms. They underuse the budget and lack the goal-directed persistence of a human engineer. The result is localized changes over primitive cores: passive planners, append-only context, permissive verifiers.
Open questions after reading. The main experiment fixes a single policy (DeepSeek-V4-Flash); the cross-policy check covers only two policies and two evolvers, which is thin. The human-engineered baseline is a composite (MiroFlow + Stirrup + Claw-Eval), not necessarily the strongest single hand-built harness, so using it as a ceiling is weak. Each model is run once with no variance, so small gaps like 46.3 vs 45.8 are not necessarily stable. And Office stagnating across the board is left undiagnosed: it is unclear whether the tasks are inherently harness-insensitive or current models simply cannot build procedural workflows.