NVIDIA study says SOAP and Muon beat AdamW on multi-billion-parameter LLM pretraining
sudoraohacker · x · 2026-07-24
NVIDIA researchers present SOAP, Muon, and beyond: a study on scaling LLM pretraining optimizers.
- The paper finds that higher-order optimizers like Muon and SOAP can converge faster than AdamW, but practical adoption has been limited by compute cost and numerical stability.
- The authors propose algorithmic fixes for SOAP at large batch sizes, including per-step QR orthogonalization and improved preconditioning to remove loss spikes and stabilize training.
- They run a unified empirical comparison of SOAP, Muon, and AdamW with matched update-RMS, and report that SOAP and Muon consistently outperform AdamW at the scales tested.
- In experiments on multi-billion-parameter models trained on trillions of tokens, the optimizers remain stable even at batch sizes up to 100M tokens for next-token prediction, where AdamW degrades.
- They also introduce a layer-wise distributed optimizer compatible with Megatron-LM and release code via NVIDIA-NeMo/Emerging-Optimizers.
More from Infra
- Broadcom Reportedly Lends Money to OpenAI for Chip NRE, Solving Startup Funding Gap — MikePFrank · 2026-08-26
- VPS-Audit: Lightweight Open Source Script for Automated Server Security and Performance Auditing — tom_doerr · 2026-08-26
- Optimizing Qwen 27B on Single RTX 3090: Quantization and Context Settings — randomjapaneselearn · 2026-08-26
- Apple's Mini/Studio refresh and OpenAI's Jalapeño both put pressure on Nvidia — Stratechery · 2026-08-26
- Spain plans stricter rules for data centers on water, energy, and security — Polymarket · 2026-08-26
- TorchMorph: CUDA-Accelerated Morphological Transforms for PyTorch — kornia_foss · 2026-08-26