CMU study: at large pretraining data scales, AdamW beats Muon after fine-tuning
AdtRaghunathan · x · 2026-11-04
CMU researchers including Catherine Li, Andrej Risteski, and Aditi Raghunathan examined whether Muon's pretraining loss advantage over AdamW persists after fine-tuning.
- Muon achieves lower pretraining loss, but the fine-tuning outcome depends on pretraining data scale
- At lower data budgets, Muon dominates after fine-tuning
- Beyond a sufficiently large data budget, AdamW outperforms Muon after fine-tuning
Takeaway: optimizer advantages don't transfer uniformly across scales — pretraining data volume reverses the Muon–AdamW fine-tuning gap.
Related event: CMU Paper: With Enough Pretraining Data, AdamW Beats Muon After Fine-tuning(4 posts)→
More from Research
- Vector and Waterloo's Chrysalis makes students teach the AI to finish assignments — VectorInst · 2026-11-04
- Control test with zeroed weights debunks claim that models read trained weights — kevrussell · 2026-11-04
- Neuro for AI & AI for Neuro workshop at AAAI 2027 opens call for papers, due Nov 20, 2026 — naturecomputes · 2026-11-04
- RLM researcher defends Weight Space Programming against 'spiteful' backlash — irombie · 2026-11-04
- Multimodal benchmarks should undergo blind filtering by default, researcher argues — cwolferesearch · 2026-11-04
- Larger tokenizers suffer more multi-epoch regret, small vocabs survive repeats — SonglinYang4 · 2026-11-04