Temporary Weight Decay in Muon Pretraining Ends Worse but Finishes 0.028 nats Better
ziv_ravid · x · 2026-11-19
A Meta team (first author Md Rifat Arefin) published an arXiv paper on the delayed effects of temporary weight decay in Muon-trained decoder-only Transformers (162M–3.4B params).
Key findings
- Applying weight decay for the first 44% of training then removing it looks worse than no decay at removal, but ends better by the final training budget.
- Transferring a 162M-selected schedule to 1.4B without retuning: 0.26 nats worse when removed, overtakes no-decay only after 73% of training, and finishes 0.028 nats better while beating the best sampled constant-decay baseline.
- For models up to 1.4B, moving the decay window from early to late training reverses its final effect; at 162M, window duration changes final loss even at roughly matched total shrinkage.
Mechanism
- Early decay leaves a persistent weight-norm deficit, keeping relative steps larger after the intervention ends.
- An approximate norm-dynamics recurrence captures the slow convergence; fixed-norm replays reproduce the performance ordering. Norm-reset experiments at 162M/405M show smaller norms at window end explain most of the final improvement.
Caveats: mostly single-seed; no help for AdamW+ReLU² at 405M; downstream evals show no clear win; the why remains open.
More from Infra
- Bill Ackman slams Nvidia board for 'major governance failures' over Jensen Huang succession — Polymarket · 2026-11-19
- 5060 Ti 16GB vs 5070 12GB: Reddit debates VRAM vs speed for local AI use — BuckinBronco999 · 2026-11-19
- rakyll is bringing agentic compute and platforms to Kubernetes, asks what to build — rakyll · 2026-11-19
- Arm CEO: NVIDIA Partnership Unchanged; AI Boom Resembles 1880s Railroad Buildout — BenBajarin · 2026-11-19
- Blackstone's Jon Gray: AI Is a 'High Bubble Alert' but Negativity Has Gone Too Far — BenBajarin · 2026-11-19
- YC-backed Tilion launches browser infra for agents, claiming #1 in stealth and speed — joshalbrecht · 2026-11-19