Study: the best optimizer depends on batch size, rankings reverse even after retuning

ChengleiSi · x · 2026-10-15

Researchers from Princeton, Stanford and UCSD show that optimizer scaling rules are less general than claimed, and optimizer rankings can reverse across batch sizes even after extensive hyperparameter retuning.

Key findings:

• The best LR scaling rule depends on task and target batch size; competing theories disagree
• On a noisy quadratic, SGD wins at small batches while Newton wins at large batches
• In SignSGD toy models, LR scaling depends on curvature relative to gradient noise
• In LM pretraining, keeping only the sharpest 0.001% of matrix directions from small-batch updates removes up to 59.5% of the error from scaling batch size
• Plots show Lion trails badly on FineWeb validation loss from 128K to 2M batch, while SOAP/Shampoo/Muon/Adam stay close

Original post →

More from Research

• Microduck robot runs onboard SLAM with just 64 pixels of depth, 4.8 cm median error — chris_j_paxton · 2026-10-15
• UIUC paper explains sparse LLM post-training updates through the lens of Fisher information — tokenbender · 2026-10-15
• Shannon's 1971 recollection: von Neumann suggested 'entropy' because 'no one really knows what entropy really is' — eigenron · 2026-10-15
• Two-person team trains HobbyLM MoE in ~304 GPU-hours, 100-400x less compute than peers — msharmas · 2026-10-15
• AReA paper brings adaptive receptive-field attention to high-res diffusion transformers — Apprehensive_Sky892 · 2026-10-15
• ACCRUE: continual off-policy RL lets robots learn new tasks in 4-12 minutes with 98.3% skill retention — coallaoh · 2026-10-15

Research channel →