Princeton/Stanford study: optimizer rankings reverse as batch size changes

_arohan_ · x · 2026-11-13

Research by Xingyu Dang (Princeton), Kaiyue Wen (Stanford) and Sadhika Malladi (UCSD), "The Best Optimizer Depends on Batch Size", shared by arohan. Key findings:

Scaling rules don't transfer consistently across settings: the best rule depends on task and target batch size;
Optimizer rankings can reverse as batch size changes, even after extensive hyperparameter retuning — fixed-batch benchmarks like AlgoPerf and NanoGPT miss this;
In a SignSGD toy model, LR scaling depends on curvature relative to noise; on a noisy quadratic, SGD wins at small batches and Newton at large ones;
In LM pretraining, keeping small-batch updates in the sharpest 0.001% of matrix directions removes up to 59.5% of the error from batch-size scaling;
Tested with Muon across competing scaling rules.

Original post →

Princeton/Stanford study: optimizer rankings reverse as batch size changes

More from Research