EvoHarness-RL: Learning Self-Evolving Runtime Harness for Long-Horizon LLM Agents
Xuying Ning, Dongqi Fu, Tianxin Wei, Hanqing Zeng, Yuanchen Bei, Bingxuan Li, Zihao Li, Qifan Wang, Xiang Shen, Yifan Wu, Jiayi Liu, Hong Li, Yinglong Xia, Xiangjun Fan, Hanghang Tong, Jingrui He
COLM 2026
cs.LG, cs.CL
2026-08-06
EvoHarness-RL trains an agent to read and update external state (belief/progress/experience) via SFT and cost-aware GRPO; Qwen3-8B hits 96.9% on ALFWorld, matching Claude Opus.
In long-horizon tasks an LLM agent has to lean on external scaffolding to remember things: where objects are, which step it is on, which trick worked last round. Today that memory and tooling is stuffed into prompts, or read/write decisions are governed by hand-written heuristics. The policy for what external state to keep and when to touch it is largely hand-engineered.
This creates two coupled problems. One is forming useful state out of noisy interaction traces; the other is deciding at runtime whether the cost of reading or writing that state is worth it right now. The paper bundles both into what it calls harness policy learning: learn an external-state policy offline, then let the agent maintain that state itself online.
EvoHarness-RL collapses heterogeneous external components into three policy-facing states, collectively BPE:
The exposed action set is just four meta-actions: track[object] reads Belief, commit[subgoal] writes Progress, recall[query] retrieves Experience, and note[insight] records a new insight. The payoff is that domain differences get hidden behind one environment adapter, so the policy sees a uniform action space.
Training is two-stage. Supervised fine-tuning first teaches the base model the action semantics and how to build useful state from expert demonstrations. Then a cost-aware GRPO stage optimizes a trajectory-level reward:
R = Rsucc + λeff · Reff + λdiv · Rdiv − λspam · Rspam − λinv · Rinv
Success gives 10×1[solved]; an efficiency term Reff = max(0, 1 − |τ| / Tmax) rewards shorter trajectories; plus an action-diversity bonus and spam/format penalties. The efficiency term is the lever: it pushes the agent not to touch external state every step, only when it pays.
Main results on ALFWorld (text household tasks, 140 seen tasks, 6 families):
| Method | Backbone | Avg success |
| ReAct | Qwen3-8B | 47.9% |
| GRPO | Qwen3-8B | 65.6% |
| SkillOS | Qwen3-8B | 80.2% |
| SkillRL | Qwen2.5-7B | 89.9% |
| EvoHarness-RL | Qwen3-8B | 96.9% |
That is a +49.0 absolute gain over ReAct. By stage: inference-time harness only (Base) 56.4%, plus SFT 68.6%, plus GRPO 96.9%. Both stages pull weight.
Bolting the inference-time harness onto frontier models lifts the weak ones most: GPT-4.1 47.9% to 70.0% (+22.1), GPT-5 60.7% to 85.0% (+25.7); the near-saturated Claude Opus 4.5 goes 96.4% to 98.5% (+2.1).
On unseen tasks, ReAct is 50.0% and EvoHarness-RL 86.6%. Strip any one BPE component and Base drops from 56.4% to between 48.6% and 50.7%; Experience is the one it misses most.
The headline claim: what helps a long-horizon agent is not bigger memory or stronger tools, but a trainable policy over external state. An 8B model matches Claude Opus on a benchmark where frontier models are already near the ceiling, and it gets there by turning memory use itself into something you train.
A second phenomenon, harness annealing, stands out. Early in training the agent calls the harness constantly, like a novice on training wheels. As GRPO proceeds, recurring patterns get internalized into the policy, call counts fall to roughly one per episode, and the agent shifts to on-demand access. The direction is the opposite of adding more RAG: externalize first, then pull back.
Validation runs on a single environment, ALFWorld, which is the main weakness. ALFWorld is text household tasks; the authors concede it favors Experience, and another domain might hinge on Belief or Progress. BPE is a functional abstraction, and deploying it still needs a per-domain environment adapter, so it is not plug-and-play.
The 8B-matches-Claude-Opus framing needs an asterisk. On ALFWorld the frontier models already sit around 96%, so the gap is small to begin with; 96.9% is a number on one narrow task, and open-ended long-horizon settings like coding or web agents are untested.
Training needs expert demonstrations for SFT plus GRPO, so the two-stage pipeline is not cheap. The paper has no formal limitations section; the points above come from reading it critically.