OpenForgeRL: Train Harness-native Agents in Any Environment
Xiao Yu, Baolin Peng, Ruize Xu, Hao Zou, Qianhui Wu, Hao Cheng, Wenlin Yao, Nikhil Singh, Zhou Yu, Jianfeng Gao
cs.AI, cs.CL
2026-07-24
OpenForgeRL fixes the train-deploy mismatch in agents, where simplified research harnesses differ from real deployment ones. A lightweight proxy hijacks the real harness's LLM calls into an RL framework, and with few RL tasks lifts a Qwen3-30B agent across four benchmarks.
Today's coding agents (Claude Code, Codex, OpenClaw) all run inside a complex inference harness: multi-turn reasoning, nested tool calls, multiple processes. But when researchers train agents, they usually reimplement a simplified harness to make RL tractable, then deploy against a completely different real one. That train-deploy mismatch erodes training gains and blocks studying how the harness itself shapes agent behavior.
OpenForgeRL (Microsoft Research, Columbia, Dartmouth) trains agents directly inside the real harness, so what is trained matches what is deployed.
Two difficulties: the real harness turns inference into a stateful multi-process flow with nested tool calls that training stacks cannot natively express, and harness rollouts need containerized environments that cannot be co-located with training nodes at scale. OpenForgeRL's answer is three pieces.
The RL algorithm is GRPO, with a terminal reward (task success) and gamma=1.0. The pipeline is harness-agnostic in principle: any harness, any environment.
Two agents, little data, clear gains.
Coding agent (OpenForge-Claw, Qwen3-30B-A3B):
| Benchmark | SFT | SFT+RL |
| ClawEval pass³ | 21.7 | 31.7 |
| QwenClawBench pass@1 | 32.1 | 33.7 |
| MCPAtlas pass@1 | 23.6 | 28.1 |
GUI agent (OpenForge-GUI, Qwen3-VL-8B):
| Benchmark | SFT | SFT+RL |
| OSWorld-Verified | 34.4 | 37.7 |
| OnlineMind2Web | 57.4 | 63.0 |
| WebVoyager | 61.5 | 72.3 |
The 8B GUI agent reaches 72.3 on WebVoyager, beating the similar-size UI-TARS-1.5-7B and matching or exceeding much larger models like OpenCUA-32B and Qwen3-VL-235B. Data is small: coding uses 892 SFT trajectories plus 343 RL tasks; GUI uses 795 SFT plus 252 RL tasks.
One important finding: training transfers across harnesses. A model trained only on ZeroClaw gains 3.3 on the unseen OpenClaw and 4.6 on Codex. RL mainly improves agent reliability (self-verification, tool coverage, multi-step plan completion), with smaller gains on single-step capability.
It turns "train agents in the real harness" from a closed-lab privilege into something the open community can do. The research value is threefold: train in any harness and any environment with no train-deploy mismatch; let the community systematically study how harness choice shapes agent behavior for the first time; and win reliable cross-benchmark gains from little data.
For teams training agents, it is a reusable engineering pattern: no more reimplementing a harness for RL, just hijack the real harness's calls.
Error recovery stays weak, with limited gain even after RL; the authors think it needs dedicated data or methods beyond RL alone. Harnesses differ widely in difficulty: OpenClaw is far harder to train than better-aligned, simpler harnesses. Partial rollouts from network failures, harness crashes, or timeouts are discarded rather than given partial credit, losing training signal.
More broadly, all the gains are on home or adjacent benchmarks, with no direct comparison to top closed agents (the models behind Claude Code, the most advanced Codex). The "big gains from little data" conclusion is bounded by benchmark ceilings, and these benchmarks are still some distance from genuinely complex real tasks.