Notes on SOAP, Muon and KL-SOAP: AdamW hits a large-batch wall at 72B scale

tokenbender · x · 2026-07-24

These notes summarize the paper “SOAP, Muon, and Beyond: Pushing LLM Pretraining Scales”.
Main claim: the so-called critical batch size is not only a property of model and data; it is also an optimizer property, and AdamW breaks down when gradients get too clean at very large global batches.
The author explains why synchronous pretraining at scale makes small batches communication-bound, and why large batches are economically attractive but expose optimizer limits.
According to the notes, Muon and KL-SOAP beat AdamW at 72B scale, with KL-SOAP slightly ahead, while Muon uses less state than AdamW.
The included diagrams argue that in MoEs, sparse experts may stay stable, but dense/shared parameters can become the bottleneck under huge batch sizes.

More from Research