Muon may help agentic RL post-training

Kai Ruan · hf · 2026-07-20

Muon can improve agentic RL post-training

The paper studies whether Muon helps sparse-reward agentic reinforcement learning, where its benefit was previously unclear.

On ALFWorld with Qwen2.5-0.5B-Instruct, the authors compare vanilla Muon against AdamW under matched single-seed runs.
With GiGPO, applying Muon only to hidden weight matrices raises final-window validation success from 0.290 to 0.546 (+88%). High-rate AdamW controls do not retain post-update success.
The effect depends on both the advantage estimator and the learning rate.
At 3e-5, Muon improves GRPO from 0.161 to 0.268.
For GraphGPO, the gap narrows near saturation at 1e-5, while Muon reaches 0.901, improves normalized validation AUC from 0.399 to 0.556, and hits 0.5 and 0.75 success 30 and 60 updates earlier.

The authors conclude that Muon can help agentic RL, but multi-seed and cross-task validation are still needed.

More from coding & agent