New paper: delayed effects of temporary weight decay in Muon pretraining

ziv_ravid · x · 2026-11-19

Paper "Delayed Effects of Temporary Weight Decay in Muon Pretraining": applying weight decay for the first 44% of Muon pretraining then removing it makes the model worse than no decay at removal, but better by the end of training. Early decay keeps weight norms small, and with Muon smaller norms mean larger effective steps for the rest of training — the cost shows up immediately, the benefit arrives late.

Related event: Temporary Early Weight Decay in Muon Pretraining Shows Delayed Payoff(8 posts)→

Original post →

More from Research

Research channel →