Molt: A Pure PyTorch Native Agentic RL Framework

青稞AI · wechat · 2026-07-16

Molt is an agentic-first, pure PyTorch Native reinforcement learning framework designed to make agentic RL "code-readable, modifiable, and scalable to massive MoE models."

Core Design

The core RL code is around 9,000 lines, and the entire runtime relies solely on Ray + vLLM + NVIDIA AutoModel/FSDP2, with no Megatron dependency.
Treats agents as Python programs: agents, reward functions, tool calls, LLM-as-judge, and VLM environments can all be directly integrated.
Asynchronous training pipeline: vLLM handles Rollouts, PyTorch/FSDP2 handles training, and Ray manages orchestration and queuing.

Key Engineering Capabilities

Supports multi-turn tool calling, LLM-as-judge, multimodal VLMs, SFT, and various RL algorithms (REINFORCE, RLOO, GRPO, PPO, etc.).
Emphasizes training-inference consistency: uses vllmkl and importance sampling for bias correction to handle logprob mismatches caused by async rollouts.
Provides RouterReplay and RouterFreeze for MoE scenarios to prevent expert routing inconsistencies between training and inference.
Supports fully async rollouts, partial rollouts, CPU offload, and parallelism strategies like TP/EP/CP.

Scale and Positioning

Officially claims to have validated MoE training ranging from Qwen3.5-397B to GLM-5.2753B.
The author compares it with verl / slime / OpenRLHF, highlighting the difference: lighter weight, pure PyTorch, yet still capable of handling massive models.

The post concludes with command-line quickstart examples and links to the repository and technical report, making it a great reference for those working on agentic RL or large-scale post-training.

Original post →

Molt: A Pure PyTorch Native Agentic RL Framework

More from coding & agent