CMU Paper: With Enough Pretraining Data, AdamW Beats Muon After Fine-Tuning
AdtRaghunathan · x · 2026-11-04
rosinality shares a new CMU paper (arXiv:2611.03633) asking whether Muon's optimizer advantage reverses at scale, posting the title page and key figures.
Key findings
- Muon beats AdamW at minimizing pretraining loss, but beyond a sufficiently large pretraining data budget, AdamW outperforms Muon after fine-tuning.
- Larger models flip at smaller data budgets: on StackMathQA the crossover TPR is 320 for 30M models but only 10 for 600M.
- At small budgets Muon forgets less and has a better learning-forgetting tradeoff; at larger budgets the trend reverses.
Framework: treating fine-tuning as a perturbation of outputs, the authors introduce the margin-to-spread ratio (how far the correct logit sits above wrong ones, relative to the spread of wrong logits). Muon has the larger ratio at small data budgets, AdamW at large ones; in linear associative memory they prove gradient descent's ratio overtakes Muon's as data grows.
Experiments span OLMo-style 30M–600M models, 150M–15.4B pretraining tokens, 600 pretraining and 29,000 fine-tuning runs.
More from Infra
- SpaceX's Starfall sells orbital factory space: 1,000 kg payload, always-on Starlink — heyshrutimishra · 2026-11-04
- AI Market Weirdness: Grok Wind-Down, Intel Down 42% in 4 Days, OpenAI Valuation Cut — JensHonack · 2026-11-04
- Formalizing PTX in Lean to formally verify GPU kernels written by AI agents — gharik · 2026-11-04
- Meta seeks $25B in bonds to fund AI buildout after S&P downgrade — ns123abc · 2026-11-04
- Snowflake launches Cortex Training: serverless fine-tuning for open-weight LLMs, claims up to 61% cost savings — StasBekman · 2026-11-04
- RISC-V Day Tokyo 2026 unveils agenda with Google TPU v3 on display and 2nm AI accelerator talks — DavidBennett__ · 2026-11-04