NVIDIA says AdamW hits a scale ceiling as SOAP and Muon beat it on trillion-token runs
omarsar0 · x · 2026-07-27
NVIDIA reports a new study on pretraining optimizers for large LLMs, arguing that AdamW may have a scaling ceiling.
Key findings:
- At batch sizes up to 100M tokens for next-token prediction, SOAP and Muon keep training stable and quality high while AdamW degrades.
- The team fixes SOAP instability at large batch sizes with per-step QR orthogonalization and better preconditioning.
- Muon’s orthogonalization quality is measured empirically rather than assumed.
- On multi-billion-parameter models trained on trillions of tokens, SOAP and Muon consistently outperform AdamW.
- They also introduce a layer-wise distributed optimizer compatible with Megatron-LM that preserves the convergence gains while balancing memory and hiding communication overhead.
The paper positions optimizer choice as a real scaling bottleneck, not just a theoretical detail.
More from Infra
- AI storage meme says the “marketing answer” is 5.3, but the real benchmark is still undefined — JoshuaJBouw · 2026-07-27
- Local LLM builders ask whether RTX Ada workstation cards are worth tracking — egudegi · 2026-07-27
- Guide breaks down how to cut Microsoft Fabric capacity costs on Azure — adnan_hashmi · 2026-07-27
- Introductory guide explains the `sempy.fabric` package for Microsoft Fabric — adnan_hashmi · 2026-07-27
- Voice AI’s real call cost is more than minutes: STT, TTS, SIP and retries — decant338 · 2026-07-27
- Micron and Meta paper says Spark can slow down 38× when shuffle spills to SSD — dr_alphalyrae · 2026-07-27