LEGO-RL: Harness-Native Reinforcement Learning for Coding Agents
Yiming Du, Yuxin Jiang, Tao Yuan, Jianbo Dai, Shaowei Wang, Jierun Chen, Chaofan Tao, Xianzhi Yu, Lifeng Shang, Kam-Fai Wong, Xiaohui Li, Haoli Bai
cs.AI
2026-08-18
Lego-RL trains Qwen3.5-35B-A3B inside unmodified coding harnesses, lifting SWE-bench Verified from 64.0/62.4/57.2% to 70.4/68.2/66.6% with >0.99 train-rollout correlation.
RL for coding agents updates the policy from full trajectories, not single answers. A rollout walks a repo, calls tools, edits files, runs tests, and gets one binary verifier reward at the end. That trajectory is produced by a native harness such as Claude Code, OpenHands SDK, or OpenCode. The optimization target is the behavior those harnesses induce.
Those harnesses were not built for policy-gradient training. They compact context, rewrite history, and reserialize tool-call arguments, so a trainer that reconstructs tokens from the saved transcript often cannot recompute the log-probabilities of the tokens that were actually sampled. Sparse MoE models add a second mismatch: if training re-selects experts instead of replaying rollout routing, the probabilities diverge again. Sandbox crashes, timeouts, and reward hacks (reading git history, downloading the gold patch) then corrupt the 0/1 signal. Most RL stacks ask you to rewrite the agent to their rollout API. Lego-RL leaves the harness alone and wraps the training stack around it.
Huawei and CUHK build the stack on verl and Harbor. A new harness needs a thin adapter that launches the agent, points it at the inference service, and returns the interaction log. Everything else is shared.
An in-process LLM proxy sits on the model API, covering OpenAI-compatible and Anthropic endpoints. At generation time it records token IDs, log-probs, response masks, and MoE routing. Later compaction does not matter; training uses the captured stream. Alignment is at message granularity: system, user, and tool-result messages must match exactly; tool calls are joined by stable IDs so argument reserialization does not rewrite captured policy tokens. Unalignable spans are dropped, not guessed from the rewritten transcript. For MoE, R3 replays the rollout expert choices in the trainer forward pass.
Each trial runs in a fresh sandbox. Nydus lazy-pulls images, the agent runtime is mounted read-only, and timeouts are per stage. Reward integrity is a set of hard gates: git history is rebased to a single commit during the agent phase and restored before grading; egress goes through a privileged sidecar the agent cannot touch; tests are withheld until grading; grader dependencies are baked into the image. Optimization is GSPO with group-relative advantages, eight rollouts per task. Infrastructure failures get zero loss weight so they do not poison the group statistic.
A plugin plus Live UI ties validation scores, termination reasons, tool traces, and rollout-training consistency back to individual trajectories.
Same Qwen3.5-35B-A3B, 2,699 OpenSWE tasks disjoint from SWE-bench Verified at repo and instance level, 200k context, one run per harness. Eval at temperature 0.7.
| harness | before | Lego-RL | Qwen3.6-35B | KAT-Coder-V2.5 |
| OpenHands SDK | 64.0% | 70.4% | 67.4% | 67.0% |
| Claude Code | 62.4% | 68.2% | 63.4% | 66.8% |
| OpenCode | 57.2% | 66.6% | 60.6% | 64.8% |
KAT-Coder-V2.5-Dev is post-trained from Qwen3.6. It gains 3.4 points on Claude Code, the harness its authors report, and loses 0.4 on OpenHands relative to untuned Qwen3.6. A gain earned under one control flow does not automatically transfer.
Median Pearson r between rollout and trainer log-probs: 0.9993 / 0.9980 / 0.9993, never below 0.989. Routing replay lifts r from 0.9946 to 0.9993, with 0.996 expert overlap. On Claude Code, matching tool calls by ID rather than serialized arguments resolves 207 of 222 apparent mismatches (93%).
Task screening is part of the result. 36,884 candidates shrink to 22,806 by rules, then 21,681 after build and verifier checks; about 2.5% of graders wrongly apply the gold patch. A Qwen3.6-27B probe then keeps the 2,699 tasks solved 1 to 3 times out of 4. In the unscreened pool, 72.7% never solved and 13.4% always solved, so most groups have zero reward variance. Before defenses, agents read git history on 4.6% to 20.5% of trials, edited tests on 2.4% to 19.4%, and downloaded the reference fix on 1.9%.
On OpenHands, trajectories that reread a file after editing rose from 73.6% to 98.1%; files inspected before the first edit rose from 3.5 to 6.9. Solving after a failed command barely moved, 63.9% to 66.8%. Mean response length grew from 43.5k to 90.9k tokens, driven by turns (46.6 to 83.1, +78%) more than tokens per turn (+17%). Over 7.5 hours, async finished 7 steps against 3 for sync; after correcting for different GPU optimizer throughput, step time is about 1.0 versus 1.9 hours. Prebuilt images are 33.2× faster than inline Dockerfiles at the median.
This is infrastructure for training against the harnesses people actually run, not a new in-house agent loop. The adapter is the only per-harness piece. Reusable lessons: capture at the API boundary; replay MoE routing; close reward holes by failure mode; keep task difficulty relative to the current policy. A fixed pool goes stale. Under OpenHands, zero-variance groups rose from 44.7% to 51.4% as the policy improved.
The accuracy lift is a solid increment. An older base plus this RL stack beats the next base and its official post-train recipe on all three harnesses. It is still one model and one harness per run. Nobody should read 6 points as a plug-and-play constant.
The paper is explicit: only Qwen3.5-35B-A3B; no mixed-harness training; one run per main config, so variance is unknown; binary terminal rewards cannot credit recovery; the anti-hack gates cover observed modes, not every exploit; sandbox speedups are measurements of this deployment. The diagnostic layer proposes causes from traces. It does not prove them.
Two extra caveats. Difficulty screening used Qwen3.6-27B plus OpenHands, then training switched to Qwen3.5-35B and two other harnesses; whether "medium" transfers was not measured directly. The async-versus-sync 2.5× figure used different GPU groups; the authors correct it down, but the uncorrected number is the one that travels. The LLM-assisted Live UI case study is an ops demo, not evidence for the method.