Weight norms never recover after early decay: 60-66% at training end

ziv_ravid · x · 2026-11-19

At 1.4B, weight norms sit at 12-16% of the no-decay baseline when decay is removed and only recover to 60-66% by the end of training — the persistent norm suppression underpins the larger effective steps and final loss gains.

Related event: Temporary Early Weight Decay in Muon Pretraining Shows Delayed Payoff(8 posts)→

Original post →

More from Research

Research channel →