4,000+ Models Show Muon's Speedup Shrinks to 10% at 1.2B Scale Over AdamW
_arohan_ · x · 2026-11-13
A new paper, 'Fantastic Pretraining Optimizers and Where to Find Them,' trains over 4,000 models to find the fastest pretraining optimizer. Key finding: claimed 2× speedups over AdamW are unlikely — beware under-tuned baselines and limited scale. Muon, for instance, delivers 40% speedup below 0.5B but only 10% at 1.2B (8× Chinchilla-optimal). Commenters conclude there's essentially no alpha left among optimizers.
More from Research
- Agent-Generated Heatmap Shows Inkling 2 Small's Split Q/K/V Queries Diverge in Later Layers — stochasticchasm · 2026-11-13
- Study of 135 model releases: AI labs barely share safety benchmarks, top one used by only half — ChenhaoTan · 2026-11-13
- Stanford Paper Shows Full Disclosure Can Hurt LLM Trading Agents Across 10 Frontier Models — shangbinfeng · 2026-11-13
- UMI diffusion policy hits 100/100 cube-stacking success with slowed demos — DominiqueCAPaul · 2026-11-13
- Wharton's Generative AI Lab open-sources Primer, an agent system turning teacher talk into classroom simulations — emollick · 2026-11-13
- Why organizations should be the focus of an AI research agenda — soumitrashukla9 · 2026-11-13