New research: standard SGD matches AdamW for LLM RL training, with far less memory overhead

zhaoran_wang · x · 2026-09-14

A new study challenges the default assumption that AdamW is required for training LLMs with reinforcement learning.

Key findings:

During RLVR, vanilla SGD performs on par with AdamW while being far more memory-efficient, since AdamW must store both first and second gradient moments.
The authors argue RL is a fundamentally different training regime from SFT: the variance of the second moment in RL is roughly 22x lower than in SFT, so most parameters share nearly the same effective step size, making adaptive learning rates largely unnecessary.
Momentum also matters far less than assumed, suggesting RL fine-tuning can drop AdamW entirely for a sparser, cheaper optimizer setup.

More from Infra