Decay schedule tuned at 162M transfers to 1.4B, overtaking no-decay at 73%

ziv_ravid · x · 2026-11-19

A schedule picked on the 162M model transfers to 1.4B without retuning: it passes the no-decay run at 73% of training and ends ahead of the best constant decay tried. Setup: decoder-only Transformers, 162M to 3.4B, trained with Muon, weight decay on for an early window then off.

Related event: Temporary Early Weight Decay in Muon Pretraining Shows Delayed Payoff(8 posts)→

Original post →

More from Research

Research channel →