EVOHARNESSBENCH: Can Your Agents Keep Pace with an Evolving Harness?
Zixuan Ke, Vaidehi Patil, Haizhou Shi, Yang Li, Ye Liu, Sarath Shekkizhar, Anurag Koul, Jiayu Wang, Xuan Phi Nguyen, Semih Yavuz, Mohit Bansal, Shafiq Joty
cs.MA, cs.CL
2026-09-03
EvoHarnessBench grows tools, skills, and agents across 17 streams (802 tasks). Catalog growth alone forgets solved work (-34.7% on agents); adaptation and retention trade off.
An LLM agent is more than a model and a prompt. It sits inside a harness: the tools it can call, the skill write-ups it can retrieve, and the specialist agents it can delegate to. Salesforce's Agentforce catalog and OpenAI's public skills repo keep adding capabilities. Existing benchmarks usually freeze that harness and put non-stationarity in the task stream, or they score skills the agent grew itself. In production, the thing that moves is the externally supplied capability surface.
Two problems split apart once the harness grows. Retention: the tools needed for old tasks are still there, now buried in a larger catalog, so previously working behavior gets harder to recover. Adaptation: if the system carries memory, learned skills, or prompts across stages, those artifacts can go stale and pull execution back onto paths that worked under the old harness.
EvoHarnessBench does not invent new tasks. It takes two verifier-based suites, EnterpriseOps-Gym and Agentic Last Exam, and splits them along tools, skills, and specialist agents into 17 nested streams of 3 to 6 stages, 70 snapshots in total. 802 source tasks become 1,510 axis-specific evaluation examples across 520 executable tools, 42 latent reference skills, and 62 specialist agents.
Capabilities are released by task frequency: core first, long tail later. The harness at stage t is the union up to t, grow-only. Each task is assigned to the earliest stage where every required capability is present and at least one of them is new. At test time the system sees the full cumulative set, not an oracle subset. By the last stage, a task faces 89.0% distractor tools or 76.8% distractor agents on average. On the skill axis, stepwise procedures are rule-extracted from source prompts into a retrievable pool and matched to tasks by verifier-state keywords; the paper treats this as a heuristic, not a proof of necessity. On the agent axis, tools are bundled by the entity they operate on; 85.2% of those tasks need coordination across specialists.
Two protocols. Deployment evaluation: no persistent state across stages, so any drop is from catalog growth alone. Self-evolving adaptation: memory (Raw Memory, ReasoningBank, MemToolAgent, G-Memory, LEGOMem), prompt evolution (GEPA), or code-level harness search (Meta-Harness) may accumulate. Model weights stay frozen. Metrics are pass rate plus relative forward transfer (FWT) on new-stage tasks and backward transfer (BWT) on older tasks under the new harness. A task-specific reference exposes only the annotated required capabilities.
Systems include ReAct (GPT-5), Codex, Claude Code (Sonnet-4.6, omitted from main tables because the model and harness differ), AutoGen, and DeLM.
The three axes fail in different ways.
Tools. Giving frontier systems the full cumulative catalog lifts overall pass rate from 24.2% (task-specific) to 28.1%, and the token bill jumps: ReAct on EOG goes from 27.2M to 75.8M. MemToolAgent raises EOG pass rate from a 30.2% deployment baseline to 38.6%, with ReasoningBank at 36.9% and Meta-Harness at 35.2%. On ALE most adaptation methods sit at or below the baseline. Deployment BWT is negative; the largest tool-axis forgetting is 5.3%. GEPA and Meta-Harness can turn BWT positive, then drop FWT on ALE to -28.5% and -11.1%.
Skills. Broadening the library barely moves deployment: Codex stays at 18.9% on EOG in both the task-specific and cumulative settings. GEPA lifts that to 24.1%. Default GPT-5 almost never invokes offered skills at the last stage; task-specific GPT-5.5 reaches 82% invocation, Claude Code 15%, and task-specific GEPA gets GPT-5 to 31%. The largest skill-axis forgetting is 4.0%, milder than tools.
Agents. Absolute scores are the lowest. Codex on EOG rises from 6.5% to 8.8% with the cumulative pool, and falls from 5.3% to 4.2% on ALE. Meta-Harness reaches 18.5% on EOG, a 110.2% relative lift over the 8.8% deployment baseline. Deployment Codex on ALE hits -34.7% BWT, the worst forgetting of the three axes. Delegation precision stays near 90% across methods; required-agent recall rises from 79% without adaptation to as high as 89%. Forgotten old tasks show delegation-set drift from 13% to 25%. They stop calling specialists that used to work. Newly added, non-required agents do not systematically take over.
Agent products add tools and sub-agents every week. This paper turns catalog growth into a measurable axis. The practical split is sharp: report retention and adaptation separately, and do not trust a final-stage pass rate. Tool evolution needs useful experience extracted from a wide action space. Skill evolution first needs the model to actually retrieve skills. Agent evolution needs to keep previously successful delegation coverage intact. For tools and skills, adapting under a narrower, task-relevant harness is cheaper and sometimes better; agent adaptation benefits more from a stable full pool from the start.
This is a benchmark paper with a real operational question, not a new algorithm.
Capabilities only grow. There is no replacement or retirement, both of which show up in real skill repos. Skill-to-task matching is keyword-based; the paper says the skills need not be necessary or sufficient. Absolute pass rates are low, especially single-digit scores on the agent axis, so the ceiling may sit in the source suites and in multi-agent coordination, not only in harness evolution. Some MAS runs were dropped for compute. Claude Code is out of the main tables. Frequency-bucketed release is a qualitative stand-in for real shipping order. None of the adaptation methods update weights, so the study measures harness-level memory, not parameter-level continual learning. FWT and BWT are marked as disagreeing across domains in several plots.