Harness Scaling for Generalizable Agentic Reinforcement Learning
Fei Bai, Huatong Song, Chuan Hao, Ran Tao, Jian Yang, Bryan Dai, Wayne Xin Zhao, Ji-Rong Wen
cs.LG, cs.AI, cs.CL
2026-10-21
GRPO on 18 orthogonal harnesses lifts Qwen3-4B in-space Pass@1 by 9.8 over single-harness RL, and beats OpenClaw training on Claude Code.
An agent harness is the shell around the model: the system prompt, which tools show up, how observations return, how context is cut, and what the loop does after a failure. Agentic RL trains the policy through that shell. Typical runs lock the shell to one configuration. The task mix and the environment can both grow while the execution path stays fixed, so the policy gets sharp at one prompt, one tool schema, and one workflow, and drops once those change.
Task scaling changes what gets solved. Environment scaling changes where the agent operates. Harness Scaling changes how the same task runs in the same environment. Claude Code and OpenClaw differ at this layer. RL inside only one of them often stays inside that shell.
Work from Renmin University's Gaoling School, IQuest Research, and Beihang splits the harness into five factors: system prompt, tool interface, observation, context management, and workflow. Three realizations each give 3^5 = 243 configurations. Hamming distance counts how many factors differ.
The full grid is a poor training set under a fixed step budget. More configurations mean fewer updates each, and low-order combinations repeat. A strength-2 orthogonal design keeps every pair of factor levels equally often. Five factors at three levels have 9 pair cells, so 18 configurations put each pair in the set twice. The Rao bound only asks for N ≥ 1+5×(3−1) = 11. First- and second-order coverage is balanced, and most complete combinations never appear, which is what in-space testing uses. These 18 are white-box: every factor realization is known.
Each task is paired with K=2 configurations. An anchor is drawn uniformly from the 18, then matched to the nearest other configuration by Hamming distance. The same task sees two nearby shells on two successive steps. Anchors from other tasks cover the rest of the training set. At K=1, diversity exists only across tasks. The loss stays the clipped PPO or GRPO objective. GRPO estimates advantage from a group of samples of the same task; PPO uses a single trajectory. Harness Scaling only changes which task–harness pair is rolled out.
Backbones are Qwen3-4B-Instruct and Qwen3-30B-A3B-Instruct, trained on 13.5K ClawGym-Task workspace tasks. Eval tasks are disjoint. GRPO uses 16 tasks and 8 rollouts per step; PPO uses 128 tasks and one rollout. Training runs 300 steps at K=2. Pass@1 (one trajectory per task) is averaged over three runs, and in-space scores are also averaged over 18 held-out configurations. Out-of-space eval uses Claude Code. Black-box SFT and RL train on OpenClaw, so Claude Code is unseen for every method. Untuned overall averages, across three benchmarks and both eval settings, are 21.89 for 4B and 31.40 for 30B-A3B.
| Model and algorithm | Single-harness overall | Scaling overall | In-space average gain |
| 4B GRPO | 29.81 | 39.04 | +9.80 |
| 4B PPO | 33.91 | 36.93 | +3.57 |
| 30B-A3B GRPO | 37.29 | 42.78 | +5.21 |
| 30B-A3B PPO | 36.31 | 41.89 | +4.90 |
The overall number mixes in-space and Claude Code. The cells do not move together. On 4B GRPO, ClawGym-Bench in-space goes from 37.33 to 52.78, and Claude Code from 38.96 to 45.37. PinchBench goes from 45.60 to 56.24 in-space and from 40.95 to 51.47 on Claude Code. JobBench-Easy is harder: in-space only goes from 10.24 to 13.55, Claude Code from 5.79 to 14.83, and the absolute score stays in the teens. One cell is flat. 4B PPO on PinchBench through Claude Code goes from 47.74 to 47.82.
White-box scaling, transferred to Claude Code, beats models trained inside OpenClaw. Under GRPO:
| Training | 4B average | 30B-A3B average |
| OpenClaw black-box SFT | 35.08 | 38.08 |
| OpenClaw black-box RL | 33.38 | 36.28 |
| White-box Harness Scaling | 37.22 | 41.20 |
On 30B PinchBench the gap is 59.78 versus 48.28 for black-box RL. Black-box RL also lands below black-box SFT on the average.
How the set is built matters more than how many shells are in it. Three harnesses with first-order coverage already add 11.43 and 4.84 on ClawGym (in-space and Claude Code) and 3.44 and 7.86 on PinchBench, relative to a single harness. The structured 18 add another 4.02 and 1.57, plus 7.20 and 2.66. A random 18 scores 47.90 and 42.78 on ClawGym, against 52.78 and 45.37 for the orthogonal set; on PinchBench the random scores are 51.28 and 48.74, against 56.24 and 51.47. Training on all 243 under the same budget, including the white-box eval configurations, still loses to those 18. The main text does not give the absolute score for that 243-run.
Moving K from 1 to 2 still helps. Moving it to 4 does not. K=2 is the best of the settings they tried. Repeat-2, which reuses one harness on the second step, is mixed against K=1 and consistently behind an actual neighbor swap. Random assignment instead of the Hamming neighbor drops ClawGym by 2.32 and 2.20, and PinchBench by 3.04 and 1.24.
Of six behaviors, state tracking, workflow compliance, and error recovery move the most. Observation reading moves the least. On 30B GRPO ClawGym, the original training harness goes from 59.82 to 59.63, down 0.19. In-space goes from 48.12 to 58.60. Claude Code goes from 46.02 to 49.62.
Shipping an agent means rewriting prompts, tool schemas, and context compression. A fresh RL run for every shell multiplies cost by the number of shells. Putting that variation into a white-box factor space, then transferring out, is the usable version of the idea.
PPO and GRPO keep their original objectives. The change is which harness is paired with which task. Eighteen configurations, two per task, 300 steps, and both model sizes and both algorithms move the same way. On the 4B model the in-space gain is 9.80 for GRPO and 3.57 for PPO.
With the budget held fixed, a balanced 18 beats a random 18 and also beats sweeping all 243. The part worth copying is pairwise balance, plus a small local swap on each task.
The five factors and three levels are this paper's own grid. Held-out configs are unseen combinations of that same family. A production harness need not land inside the 243. Transfer to Claude Code shows that some execution skill survives outside the grid. The main text fully reports only Claude Code as an external system. OpenClaw evaluation sits in the appendix and is described as the same trend, without numbers in the main text.
JobBench-Easy remains in the teens after training. The move from 5.79 to 14.83 on 4B Claude Code is real, and still far from a reliable agent. Averages rise while single cells, including 4B PPO on PinchBench, barely move.
Only two Qwen3 scales are tested, and the black-box RL baseline uses GRPO only. The orthogonal design wants the same number of levels on every factor. Permissions, sandboxes, and multi-agent roles are outside this space. The six-way behavior plot has no per-axis scores. The near-zero drop on the original harness is reported for one cell, 30B GRPO on ClawGym. Matched step budgets are not converted into wall-clock or tokens. If eighteen harnesses cost more to roll out, equal budgets are not equal cost.