ClawGym II: Exploring Black-Box RL on Agent Harness
Huatong Song, Fei Bai, Ming Yang, Renyuan Li, Jia Deng, Jujie He, Zhange Zhang, Daixuan Cheng, Yan Xing, Qi Yun, Xuxing Chen, Danyang Li, Feng Chang, Chuan Hao, Ran Tao, Jian Yang, Bryan Dai, Wayne Xin Zhao, Mingjie Tang, Ji-Rong Wen
cs.CL, cs.AI, cs.LG
2026-08-18
A Gaoling-led team shows RL can run directly inside mature opaque harnesses like Claude Code and OpenClaw: sandboxed concurrent rollouts, a serving proxy that captures model calls and rebuilds them into prefix trees, and stable PPO and GRPO optimization over the tree lift Qwen3-30A3B Pass@1 by 9.98 and 14.81 points.
Agent harnesses like Claude Code, Codex, and OpenClaw have become the operating layer for long-horizon work: system prompts, tool orchestration, context management, and retry recovery packaged into one runtime. How well a model performs inside depends heavily on whether it was trained for that harness. RL through a harness hits three hard walls: the internal control flow is opaque, single-machine rollout infrastructure does not exist, black-box execution emits fragmented, forked, redundant model-call records rather than clean trajectories, and the training and inference engines are two numerical environments whose mismatch biases gradients.
The mainstream answer is to dodge: write your own transparent agent loop and train there. The cost is that the trained model bonds to that homemade loop and loses something on real harnesses. This paper takes the opposite stance: the harness is the deployment environment, so train inside it.
The framework separates execution from optimization. On the infrastructure side, each task's initial workspace and the chosen harness are encapsulated in a temporary sandbox, provisioned at rollout start and destroyed after, enabling large-scale concurrency without interference; external capabilities mount through MCP servers inside the sandbox. On the optimization side, the harness keeps its native logic as an opaque rollout engine while the training side places a proxy at the model-serving boundary; every model call crosses it. The proxy invokes the current policy, returns responses in the protocol the harness expects, and records input tokens, generated tokens, and sampling log-probabilities. At task end a verifier scores the final workspace and emits a rollout-level reward.
Fragmented calls are rebuilt into a prefix tree: each call attaches to the node whose accumulated history is its longest prefix, and comparing a child call's input against the parent history also recovers intervening non-model content such as tool outputs. Leaf filtering does three things: drop dead leaves from inference retries or harness regenerations; discard over-branching rollouts entirely so corrupted signals never reach the gradient; exclude subagent and compaction trajectories, whose roles differ from the main task path and whose shared terminal reward would muddy credit assignment. On the tree, both PPO and GRPO are adapted so all trajectories from one rollout share its terminal reward while shared-prefix tokens count once, preventing highly branched rollouts from dominating.
Training-inference consistency gets two safeguards. The token-in-token-out discipline grafts inference-engine tokens directly onto the tree as the sole training data; the structured text the harness sees is a decoded view of those tokens and is never re-encoded, so harness reformatting cannot pollute the record. A token-level importance-sampling correction with truncation then tames outlier ratios between inference-time and recomputed probabilities. Engineering hardening covers forced timeouts, wholesale discard of sandbox failures, pseudo-streaming parsing (buffer tokens during generation, emit synthetic SSE to keep the connection alive, parse once after the turn completes, eliminating incremental-parse truncation of valid tool calls), and settling waits before trajectory assembly.
Validated on two structurally distinct harnesses: OpenClaw (general assistant) and Claude Code (long-horizon coding and terminal). After black-box RL from the Qwen3-30A3B backbone, Pass@1 on ClawGym-Bench rises 9.98 and 14.81 points respectively, with matching gains on the external PinchBench, stable across 200-400 optimization steps.
| Model (Claude Code as harness) | PinchBench | ClawGym-Bench avg |
| Qwen3-30A3B base | 54.14 | 37.06 |
| Qwen3-235A23B | 62.47 | 45.59 |
| ClawII-CC-30A3B (this work) | 71.42 | 51.87 |
A 3B-active model overtakes the 235B MoE backbone by six points. Mix-harness training (each task paired with both harnesses, randomly mixed in batch, advantages grouped per task-harness pair) matches or slightly beats single-harness models without instability. PPO and GRPO both converge, PPO with smoother entropy. Extensions to JobBench (multi-format professional workflows) and OfficeQA (document-grounded analytical reasoning) also improve. A lightweight cold-start SFT is optional but raises the starting reward and stabilizes entropy under OpenClaw.
The telling comparison: a model trained in a homemade white-box agent loop scores 59.90 in its own loop, beating the black-box-trained 51.37, but transfers to unseen OpenClaw at only 50.33, well below the 62.62 of the model trained directly in OpenClaw. White-box training acquires some general agentic ability; harness-specific interaction patterns still require the real harness.
This turns "train where you deploy" from a slogan into a reproducible engineering recipe. For model labs, harness competence can be trained on the target harness's real interactions instead of a homemade approximation, and the sandbox-plus-serving-proxy-plus-prefix-tree combination applies to any black-box system whose model calls are the sole observation point. The mix-harness result, no conflict, no degradation, means multi-harness data can train one general model.
Subagent and context-compaction trajectories are excluded from optimization entirely, listed by the authors as future work, yet these are precisely how mature harnesses handle long-horizon tasks, so only the main thread is trained. PPO forks trajectories with a simplification (independent GAE per trajectory, no cross-branch credit propagation), and the variance cost on long horizons is acknowledged but unsolved. Training tasks come from ClawGym-SynData, PinchBench shares that provenance, so generalization to real workloads lacks direct evidence. The OpenClaw branch uses a cold start while the Claude Code branch does not, making the two not fully comparable; the paper attributes part of the entropy difference to this. Rubric scoring uses GPT-5.4 as judge, so the judge's preferences seep into the numbers. Compute and sandbox scale are described only architecturally; reproduction cost is unknown.