New paper: weight decay in first 44% of Muon training hurts immediately but wins long-term
ziv_ravid · x · 2026-11-19
New paper "Delayed Effects of Temporary Weight Decay in Muon Pretraining" examines the timing effects of weight decay under the Muon optimizer: decay was applied for the first 44% of training, then removed.
Key finding: right after removal the model got worse than a no-decay baseline, but by the end of training it outperformed it. The authors' explanation: early decay keeps weight norms small, and with Muon that translates into larger effective steps for the rest of training — the cost shows up immediately while the benefit arrives late.
Related event: Temporary Early Weight Decay in Muon Pretraining Shows Delayed Payoff(8 posts)→
More from Research
- 'Physics Hasn't Progressed in 50 Years?' This New PRL Paper on Icy Friction Says Otherwise — CatAstro_Piyush · 2026-11-19
- TMol brings Rosetta-style all-atom molecular energetics into differentiable PyTorch — gottapatchemall · 2026-11-19
- ELLIS Tübingen safety team releases horizon-labs-agent-traces dataset on Hugging Face — NielsRogge · 2026-11-19
- Noam Brown confirms the math model claim; speculation points to an astra math fine-tune — haider1 · 2026-11-19
- NCME talk: generative psychometrics validates LLM-generated test items before human data — GolinoHudson · 2026-11-19
- Researchers say peer review has hit a breaking point as AI conference scale strains ICLR — LucaAmb · 2026-11-19