New paper: delayed effects of temporary weight decay in Muon pretraining
ziv_ravid · x · 2026-11-19
Paper "Delayed Effects of Temporary Weight Decay in Muon Pretraining": applying weight decay for the first 44% of Muon pretraining then removing it makes the model worse than no decay at removal, but better by the end of training. Early decay keeps weight norms small, and with Muon smaller norms mean larger effective steps for the rest of training — the cost shows up immediately, the benefit arrives late.
Related event: Temporary Early Weight Decay in Muon Pretraining Shows Delayed Payoff(8 posts)→
More from Research
- 'Physics Hasn't Progressed in 50 Years?' This New PRL Paper on Icy Friction Says Otherwise — CatAstro_Piyush · 2026-11-19
- TMol brings Rosetta-style all-atom molecular energetics into differentiable PyTorch — gottapatchemall · 2026-11-19
- ELLIS Tübingen safety team releases horizon-labs-agent-traces dataset on Hugging Face — NielsRogge · 2026-11-19
- Noam Brown confirms the math model claim; speculation points to an astra math fine-tune — haider1 · 2026-11-19
- NCME talk: generative psychometrics validates LLM-generated test items before human data — GolinoHudson · 2026-11-19
- Researchers say peer review has hit a breaking point as AI conference scale strains ICLR — LucaAmb · 2026-11-19