HarnessDev: Can LLMs Create and Evolve Their Own Agent Harness?
Yuhao Wu, Jingyuan Zhang, Jiajun Shi, Xinping Lei, Qingshui Gu, Yuxuan Zhang, Zexuan Wang, Chen He, Chen Huang, Maojia Song, Zhiyuan Zeng, Shaowen Wang, Jinkai Liu, Yunfeng Shi, Jiaheng Liu, Shen Yan, Wenhao Huang, Ge Zhang, Wenxuan Zhang
cs.SE, cs.CL
2026-09-01
Six models build a harness from a weak seed. Opus Self-Eval 67.8 vs human 86.2: writing and MLE match or beat, code and search lag. Evolution gains are small and transfer poorly.
Agent performance now lives as much in the software around the model as in the weights. That surrounding stack (the loop, tools, context, recovery, and verification) is the harness. Hold GPT-5 fixed and Terminal-Bench 2.1 jumps from 35.2% inside Terminus 2 to 49.6% inside Codex CLI. Most agent benchmarks still pick a harness and then score task answers, so the engineering that actually makes a deployment work is treated as setup, not as the thing being built.
HarnessDev scores the durable, reusable execution system instead. ByteDance Seed, with SUTD, Georgia Tech, and others, splits the job in two. Creation starts from a weak seed and must produce a complete harness. Evolution then edits that same harness from downstream execution feedback.
Every creator gets the same weak seed: it can parse config, expose low-level tools, and write logs, but it has no agent loop, decomposition, tool policy, state, verifier, retry, or stopping rule. Unmodified, it scores zero on every downstream benchmark. The creator also gets a task-family spec, permission limits, a short tutorial, and one to three development cases. It never sees the human implementation or the hidden eval set.
Evolution starts from that creator's own frozen code harness. During development it can see 100 SWE-Pro feedback tasks and all 89 Terminal-Bench tasks, with a budget of ten official full-evaluation pairs. After freeze, every official version is scored on 630 SWE-Pro instances whose scores are never returned, so adaptation and generalization are reported separately.
The six creators are Opus 4.8, GPT-5.5, Gemini 3.1 Pro, DeepSeek V4 Pro, Qwen 3.7 Max, and Seed 2.0 Pro. Development runs in Claude Code, except GPT-5.5 in Codex. Self-Eval uses the creator as executor; Unified-Eval runs every harness under Gemini 3.1 Pro. Creation builds three independent harnesses per cell and reports avg@3. Downstream coverage is 2,207 unique instances across SWE-Pro (731), Terminal-Bench 2.1 (89), MLE-bench (75), EQ-Bench3 (46), and BrowseComp (1,266).
Under Self-Eval, Opus's overall score is 67.8 against a human-engineered reference of 86.2. The gap is domain-specific.
| Method | SWE-Pro | Terminal-Bench | MLE-bench | EQ-Bench3 | BrowseComp | Avg |
| Opus 4.8 | 69.3 | 64.8 | 32.9 | 84.6 | 52.4 | 67.8 |
| Gemini 3.1 Pro | 43.6 | 68.8 | 32.4 | 74.8 | 35.2 | 55.6 |
| GPT-5.5 | 32.8 | 52.1 | 19.1 | 83.0 | 52.6 | 55.1 |
| Human reference | 80.0 | 88.8 | 24.0 | 83.7 | 92.2 | 86.2 |
Writing matches or slightly beats the reference (Opus 84.6 vs 83.7). MLE medal rates of 32.9 (Opus) and 32.4 (Gemini) both exceed the human 24.0. Search is the largest miss: BrowseComp 92.2 for the human system versus 52.6 for the best model-built harness (GPT-5.5). Code still trails, 69.3 vs 80.0. Seed 2.0 Pro lands at 22.8 overall. Of failed Data tasks, 77.8% are blamed on harness defects, so the bottleneck is often the scaffold, not the executor.
Swapping the executor reshuffles the ranking. Qwen gains 17.6 on BrowseComp and 12.9 on MLE under Gemini. Opus SWE-Pro falls from 69.3 to 33.0, and duplicate queries in its search harness jump from 10.1% to 88.2%. One Opus code harness hard-codes a 120-step cap around the original executor and nearly collapses under Gemini. MLE token use varies about nineteen-fold, and higher spend does not reliably buy a higher score. Gemini's code harness adds only 1,006 lines yet posts the best Terminal-Bench number (68.8); edit volume does not predict performance.
On Evolution, all five self-runtime lineages rise on the visible feedback pair, then shrink on held-out: Opus +4.44, mean +3.11. With a fixed Gemini executor only Opus still gains on held-out (+2.70); Qwen -1.11, DeepSeek -2.38, GPT-5.5 -10.32. Feedback and held-out move together on 34 of 64 official switches (53.1%), and only 2 of 9 declared versions are held-out optimal. Edits concentrate on control flow and tools; state is almost untouched. Across 26,679 recorded trajectories, no checkpoint event fires.
The benchmark turns forward-deployed engineering into something you can score. Weights are one place capability accumulates; the harness is another, and unlike weights it is inspectable and reusable. For product teams the takeaway is concrete. Stop reporting only the score under a frozen human scaffold. Model-built harnesses are already usable for short-form writing and ML experiments. They are not a substitute for mature human systems on code or long-horizon search. Evolution can land local patches. It is not yet a reliable CI loop.
Human references come from mixed harness-model pairs, not a common executor, so beating 100% means beating that external system, not beating humans. Evolution has one trajectory per creator-runtime cell, and the post-freeze held-out set is SWE-Pro only. The development environment stays fixed; whether an evolved harness can itself become the next development environment is untested. Four task families do not cover real deployments. Constraint audits found no cheating, but the containers are for reproducibility, not containment. Treat generated harness code as untrusted.