Muon beats AdamW in pretraining loss, but AdamW wins after finetuning at scale
AdtRaghunathan · x · 2026-11-04
A joint optimization-and-data study finds Muon achieves lower pretraining loss than AdamW, yet whether its advantage survives finetuning depends on pretraining data scale: beyond a sufficiently large data budget, AdamW outperforms Muon after finetuning.
Related event: CMU study: with enough pretraining data, AdamW beats Muon after finetuning(5 posts)→
More from Research
- Direct weight manipulation beats fine-tuning: pi0.5 hits 75% on cloth-folding — ZeeshanZiaML · 2026-11-04
- Researchers propose 'Four As' framework to untangle AI governance trade-offs — Brighter-Side-News · 2026-11-04
- Stanford's RoboArchitect: Coding Model Rewrites Robot Control Code for Under $150, Success Up to 43% — rohanpaul_ai · 2026-11-04
- LitPivot at UIST 2026: A Tool to Rework Your Scooped Research Idea Into a Better One — IanArawjo · 2026-11-04
- Idol Turbo distills image-to-3D to 5 steps: ~0.75s for up to 1M Gaussians on H200 — _akhaliq · 2026-11-04
- Classic reward-hacking cases, from CoastRunners to the asthma rule, turned into an eval checklist — ClickOk5811 · 2026-11-04