HarnessOpt-Bench: Evaluating LLMs at Harness Optimization
Varun Ursekar, Apaar Shanker, Yash Maurya, Shehab Yasser, Vijay S. Kalmath, Veronica Chatrath, Yuan Xue
cs.AI, cs.CL, cs.LG
2026-08-07
HarnessOpt-Bench has five frontier LLMs iteratively optimize an agent's harness under a fixed budget across 111 runs; the optimizer model moves gain about 1.8x more than swapping the coding harness, and native tooling gives no consistent edge.
An agent's ability depends not just on model weights but on the harness wrapped around it: prompts, tools, control flow, memory, and orchestration code. The same model behaves very differently under different harnesses. So "having an AI iteratively improve the harness" is both an important route to better systems and a demanding model capability. The trouble is that methods use their own agents, seeds, budgets, and scoring, so they cannot be compared. Scale AI turns the task into a controlled, reproducible benchmark.
The setup is built against gaming. An optimizer, an LLM paired with a coding harness, receives the target agent's seed harness, graded development and validation feedback, and a fixed evaluation budget. It edits the harness and nominates a final candidate, which is scored on a held-out test partition it never sees during search. A trusted execution environment enforces the budget, blocks held-out state, and versions every candidate for audit. The suite spans four downstream tasks (OfficeQA, BrowseComp-Plus, Terminal-Bench, GAIA) and five frontier models (claude-opus-5, claude-sonnet-5, gpt-5.6-sol, gpt-5.6-terra, kimi-k3), each under two harnesses: a shared opencode and its native one (claude-code, codex, kimi-cli), for 111 scored runs.
The strongest configuration, opus-5 with opencode, captures 0.63 normalized gain on OfficeQA, about two thirds of the headroom; the weakest, gpt-5.6-terra with codex, is indistinguishable from zero on BrowseComp and Terminal-Bench. The core result: holding task and harness fixed and swapping the model moves gain by 0.142 on average; holding task and model fixed and swapping the harness moves it by only 0.079, so the model contrast is about 1.8x larger. Native harnesses confer no consistent edge: across 20 model-by-task pairs, the shared harness wins 11, the native wins 9, with no ties. The one exception is GAIA, where both GPT models do markedly better under codex (+0.179 and +0.131) while the Claudes and Kimi are roughly even either way. On process, broader search, touching more of eight predefined harness levers, correlates with gain (Spearman +0.34 to +0.88), whereas the share of actions spent reading execution traces correlates negatively.
It makes "harness engineering as a model capability" an empirical object. Two lessons stand for agent builders. First, do not assume a model paired with its own tooling is automatically stronger; here native tooling has no consistent advantage, and which harness is best varies by model. Second, search breadth beats detail-grubbing: reading traces, the drudge work, is uncorrelated with final gain, and per-case score summaries usually localize failures well enough. The benchmark also yields a capability ladder: across five GPT releases on OfficeQA, gain rises monotonically from +0.03 to +0.49, showing it can separate model generations.
The seed harness is a task-specific prior. Three tasks are "improve a working agent" and GAIA is "build one from a stub," testing different abilities (diagnosis versus construction), yet the paper does not vary seed complexity systematically. The authors concede the benchmark is hack-resistant, not hackproof: repeated development and validation feedback can still reward strategies tuned to a fixed evaluation. Candidates are restricted to Python, and each task pins a single target model, so generalization across languages, architectures, and target models is untested. Differences among middle configurations are often smaller than round-to-round jitter, supporting tiers rather than a fine ranking.