Muon optimizer is cheap in pretraining but adds up to 54% FLOPs in fine-tuning
kastnerkyle · x · 2026-11-06
Soufiane Hayou analyzes why Muon is nearly "free" for pretraining but costly for post-training and data-constrained runs: pretraining's much larger batch sizes amortize the Newton-Schulz orthogonalization cost, while small-batch SFT does not. The chart shows Muon's extra FLOPs hitting +54% for a 700B MoE in SFT vs just +0.2% in pretraining (Dense 1B/405B show +0.10.4% pretraining vs +224% SFT). At frontier scale the core challenge is communication — distributing NS steps efficiently, where Adam is far friendlier. A detailed blogpost is forthcoming.
More from Infra
- Texas data center bill would let all permitted projects finish, just with stricter consumer rules — VraserX · 2026-11-06
- GitHub's new Git storage handles 7,200 pushes/sec from 5,000 concurrent agents — arpit_bhayani · 2026-11-06
- GitHub rebuilds Git storage for thousands of concurrent AI agents — arpit_bhayani · 2026-11-06
- OpenAI built Pando for Codex: 5x lower Git push latency, ~1PB transferred daily — sven_ai · 2026-11-06
- AI bubble debate: liquidity drying up, but are OpenAI and Anthropic really immune? — teortaxesTex · 2026-11-06
- With US data centers zoned out, builders eye orbit where no one can vote you off the planet — r0ck3t23 · 2026-11-06