Adam-to-Muon mid-training switch sparks debate, seemingly contradicting Moonlight paper
stochasticchasm · x · 2026-09-22
A technical thread on LLM training details:
- The author spots a pretraining run switching from Adam to Muon at mid-training, seemingly contradicting the Moonlight (Kimi) paper, which found keeping the same optimizer beats swapping mid-run.
- A previously unknown optimizer called Muown appears, likely a NorMuon-style row-norm variant.
- "No loss spike" doesn't rule out degraded performance — evals may just be lower.
- mxfp4 QAT during mid-training seems standard now, likely aimed at accelerating RL rollouts.
Related event: Analysts spot mid-training optimizer switches from Adam to Muon(3 posts)→
More from Models
- OpenAI has largely automated training experimental models; new model cracks 100 open math problems — kimmonismus · 2026-09-22
- MiMo-V2.6 called one of the largest open-source RL runs, now top open model — andrew_n_carr · 2026-09-22
- Dev on AI coding's biggest pain point: models are still too stupid, slow and expensive — remilouf · 2026-09-22
- GPT-6 Astra Light users say weekly quota drained in 5 hours, drawing comparisons to Fable — PratikKadam_ · 2026-09-22
- Claude refuses to push a person off a ledge, steps back 'to keep a safe distance' — repligate · 2026-09-22
- Grok Build hits 140K installs with desktop app; tests show GPT 4.6 outscoring 4.7 — PawelHuryn · 2026-09-22