Timing matters: decay in first 10% of training helps, at 60% it hurts

ziv_ravid · x · 2026-11-19

Decay over the first 10% of training helps at 162M, 405M and 1.4B, while the same window starting at 60% makes final loss worse. Practical takeaway: when tuning a regularizer that switches off mid-run, compare runs at the end of the compute budget.

Related event: Temporary Early Weight Decay in Muon Pretraining Shows Delayed Payoff(8 posts)→

Original post →

More from Research

Research channel →