NVIDIA study says SOAP and Muon beat AdamW on multi-billion-parameter LLM pretraining
sudoraohacker · x · 2026-07-24
NVIDIA researchers present SOAP, Muon, and beyond: a study on scaling LLM pretraining optimizers.
- The paper finds that higher-order optimizers like Muon and SOAP can converge faster than AdamW, but practical adoption has been limited by compute cost and numerical stability.
- The authors propose algorithmic fixes for SOAP at large batch sizes, including per-step QR orthogonalization and improved preconditioning to remove loss spikes and stabilize training.
- They run a unified empirical comparison of SOAP, Muon, and AdamW with matched update-RMS, and report that SOAP and Muon consistently outperform AdamW at the scales tested.
- In experiments on multi-billion-parameter models trained on trillions of tokens, the optimizers remain stable even at batch sizes up to 100M tokens for next-token prediction, where AdamW degrades.
- They also introduce a layer-wise distributed optimizer compatible with Megatron-LM and release code via NVIDIA-NeMo/Emerging-Optimizers.
More from Infra
- AWS on agent prompt caching: explicit + implicit patterns to slash token costs — AWS ML Blog · 2026-10-27
- Thieves spoofing GPS steal $100M+ in data center cargo since June — Ars Technica AI · 2026-10-27
- Lam Research: weekly token usage up ~250x in 18 months — Beth_Kindig · 2026-10-27
- AWS shows how to build enterprise agentic retrieval on Bedrock with CDK — AWS ML Blog · 2026-10-27
- Encrypted GPT-2 speedrun challenges FHE inference: best is 3.031 s/token, goal sub-1s — tensorqt · 2026-10-27
- Open-source Latch compiles PyTorch into training megakernels, 2.75x faster on H100 — sguada · 2026-10-27