New paper: weight decay in first 44% of Muon training hurts immediately but wins long-term

ziv_ravid · x · 2026-11-19

New paper "Delayed Effects of Temporary Weight Decay in Muon Pretraining" examines the timing effects of weight decay under the Muon optimizer: decay was applied for the first 44% of training, then removed.

Key finding: right after removal the model got worse than a no-decay baseline, but by the end of training it outperformed it. The authors' explanation: early decay keeps weight norms small, and with Muon that translates into larger effective steps for the rest of training — the cost shows up immediately while the benefit arrives late.

Related event: Temporary Early Weight Decay in Muon Pretraining Shows Delayed Payoff(8 posts)→

Original post →

More from Research

Research channel →