Molt: A Scalable PyTorch-Native Training Framework for Agentic Reinforcement Learning
Jian Hu, Huiying Li, Hao Zhang, Binfeng Xu, Yifan Zhang, Shaokun Zhang, Hemil Desai, Michael Demoret, Pavlo Molchanov, Jan Kautz, Yi Dong
cs.LG, cs.CL, cs.DC
2026-07-23
Nvidia's Molt compresses agentic RL training to about 8.6K lines, training multimodal and MoE policies in a single async loop without ever training on tokens it did not generate, matching a Megatron plus SGLang stack on Qwen3-30B-A3B and scaling to a 700B MoE.
Agentic RL research is constant algorithm change: new estimators, new pipeline stages, new rollout schemes. In mainstream frameworks each change threads through layers of trainer, distributed backend, and rollout glue, and the cost of every iteration lands on the researcher. The framework becomes more complex than the hypothesis being tested. Molt targets exactly this: shrink the framework until a researcher can hold it in their head and an AI coding assistant can read it end to end, drive down the end-to-end cost of changing algorithms, and not lose throughput to heavy state-of-the-art stacks.
Molt's trade-offs are aggressive. The framework-owned RL code is about 8.6K lines (verl about 62K, slime about 25K), with exactly one training backend (AutoModel with FSDP2) and one serving engine (vLLM), neither forked. Four core concepts: the agent is an ordinary Python program (Gym-style environment, or a ChatAgent using stock OpenAI and Anthropic SDKs), the generator does token-exact capture, the trainer is one visible training loop, and estimators and losses are pure functions.
Hard correctness invariants hold it together: token identity (sampled token IDs define trajectories), policy-version semantics (trainable tokens retain behavior-policy log-probabilities), and forward consistency (rollout and actor agree on model semantics). Engineering: a streaming pool keeps prompt groups in flight and emits training batches as groups complete; partial rollout pauses engines, broadcasts actor shards, and resumes without dropping in-flight requests; token-in/token-out (TITO) capture never leaves token space, eliminating retokenization drift; weights broadcast directly from actor to engines via NCCL. Composable parallelism: FSDP2 plus AutoModel-native tensor, expert, and context parallelism, so scaling is a configuration change, not a migration.
On footprint, Molt's full RL path is about 8.6K lines, versus about 62K for verl and 25K for slime. Engine-feature measurements (Qwen3.6-35B-A3B multimodal MoE, 2 nodes, 8 training plus 8 rollout GPUs): speculative decoding with an MTP head cuts per-step generation from 329s to 64s (5x); optimizer CPU offload drops actor peak memory from 64.7GB to 46.4GB at 18% more policytrain time.
Head-to-head: on Qwen3-30B-A3B with DAPO-Math, under a fully asynchronous disaggregated 8-plus-8 protocol against slime (Megatron-Core plus SGLang).
| Config | Step (s) | Tok/GPU/s |
| Molt (AutoModel plus vLLM) | 119.4 ± 2.3 | 461 |
| slime (Megatron plus SGLang) | 109.5 ± 10.3 | 502 |
The authors claim no superiority for either side: slime's cross-run step-time spread (102 to 121s) overlaps Molt's band (119.4 ± 2.3), and the roughly 9% mean difference sits inside the noise. For scale, a 700B MoE at expert parallelism 256 ran the full async loop end to end.
For agentic-RL researchers, this is evidence that the framework need not be the bottleneck: with under a seventh of the code of others, it matches Megatron-class throughput, and the codebase is small enough for humans and AI coding assistants to read and change end to end. It also demonstrates an engineering judgment, that scale comes from composing existing parallelism (FSDP2 plus AutoModel-native parallelism plus vLLM), not from rebuilding a distributed backend. It is open source (NVIDIA-NeMo/labs-molt) with recipes and containers.
The honest caveats: it targets research, not enterprise data-conversion layers or control planes; the throughput comparison is statistically comparable, not faster; and the 30B checkpoint exposed an upstream distributed-MoE forward mismatch, so only throughput was compared and convergence parity awaits an upstream fix.
The authors state: a narrower deployment choice (deliberate, for single-backend directness), some combinations rejected at config time (packed batches under context parallelism), no superiority claimed on throughput, and benchmark limits.
Open questions: matching Megatron is throughput parity under a single protocol (one model, one dataset, 8-plus-8 GPUs), untested across model sizes and tasks; the 700B MoE run is a it-ran demo with no convergence or quality numbers. As a lean framework, it outsources complexity to vLLM, Ray, and AutoModel, and version and bug risk in those dependencies is the other face of the 8.6K-line saving.