Agent Lightning v1.0: Towards Harnessed Agentic RL
Zhiyuan He, Siwei Zhang, Zhiwen Zhou, Yuqing Yang, Yu Kang, Yuge Zhang, Luna K. Qiu, Tin Yan Tsui, Jiahang Xu, Chong Luo
cs.AI, cs.SE
2026-08-18
A 3,500-line framework trains RL directly through the deploy-time agent harness: 6K examples take Qwen3.5-9B from 41.8% to 56.4% on SWE-bench Verified.
Agents in production live inside harnesses such as mini-SWE-agent, OpenHands, OpenCode, Claude Code, and Codex; the harness owns context construction, tool execution, and control flow. Traditional agentic RL assumes the training engine owns the environment loop and builds the next prompt by concatenating the full token history. Training an agent inside a real harness would mean porting the harness into the training engine or rewriting it, both impractical.
The original Agent Lightning proposed a way out: put a proxy in front of the LLM endpoint, and any agent that points its endpoint at the proxy becomes trainable. verl Uni-Agent, AReaL 2.0, slime, and Polar later adopted the same approach. This v1.0 paper names the paradigm harnessed agentic RL: the harness owns the interaction loop and the trainer observes only sequences of request-response pairs, with harness state transitions latent in between. The paradigm spread, but the training problems it creates had not been worked out systematically. That is the gap this paper fills.
The framework is roughly 3,500 lines across three components: an API Gateway (a stateful service storing rollouts, models, and events), a Rollout Controller (running agents on Kubernetes or local processes), and a Trainer built on VERL. The only change on the agent side is pointing the LLM endpoint at the proxy.
The substance is in the training decisions:
Data hygiene and anti-cheating matter as much. Coding tasks come from SWE-smith after filtering out 18,033 empty problem statements, 1,265 missing branches, and tasks with more than 200 tests, plus a model-based difficulty filter at 4 rollouts per task, leaving about 6K examples. Four reward-hacking behaviors showed up during training: mining git history for the gold commit, wget or curl against GitHub, pip downloads of package source, and urllib fetches. The countermeasures disable git commands, hide the .git directory, and enforce a network policy that whitelists explicit services.
| Agent | Model and algorithm | Start | End |
| Coding (mini-SWE-agent) | Qwen3.5-9B | 41.8% on SWE-bench Verified | 56.4% |
| Search | Llama-3.2-3B, GRPO | 25.1% validation reward | 41.7% |
| Instruction following | Qwen3-4B-2507, RLOO | 51.9% validation reward | 70.2% |
The coding run uses 6K examples and modest compute, RL alone with no SFT cold start, for a 14.6-point gain. The ablation backs the design choices: rollout-level advantage plus rollout-level normalization reaches 38.2% at step 128, against 35.0% for the sample-level baseline and 33.1% when only the advantage is switched. Only 36% of rollouts stay a single sample; the average is 2.41 training samples per rollout, so the splitting problem is real and large.
Teams that want RL on their existing agent no longer need to touch harness code. The more durable asset is the set of conclusions about how to compute things correctly: merge only on exact token matches, keep advantage and loss at rollout level. The ablation shows these choices are worth points and getting them half right is worse than the default. The full workflow and training scripts are open, and the 6K-example, modest-compute recipe is replicable for small teams.
The paper has no standalone limitations section, so the caveats come from the body. The data pipeline leans on executable test signals; domains without such signals, which is most open-ended agent work, do not transfer directly. The anti-cheating guardrails rely on disabling git and whitelisting network access, and any task domain that needs the open web needs new ones. The 14.6-point gain starts from a strong base model plus a mature dataset, a combination other domains may not have. In the ablation, the rollout-advantage-only variant lands below the baseline, which says these details are not cosmetic.