Temporary Early Weight Decay in Muon Pretraining Shows Delayed Payoff
Meta团队(Md Rifat Arefin一作,@zivravid为合著者并连发多条解读)发布arXiv论文《Delayed Effects of Temporary Weight Decay in Muon Pretraining》:在Muon预训练的decoder-only Transformer上,仅对前44%训练施加weight decay、之后移除,移除时模型比不加decay更差,但训练结束时反而反超,最终损失低约0.028 nats,且优于尝试过的最佳恒定decay。已确认
- 机制在162M规模被直接验证:固定范数、不加decay地重放早期decay的step-size轨迹,能得到相近收益;把两次训练的权重范数互换,收益随之迁移——单向带走96%,另一向97%。
- 在1.4B模型上,早期decay移除时权重范数仅为无decay对照的12-16%,训练结束时也只回升到60-66%,说明范数被压低后不再恢复,这正是后续等效步长变大、损失更低的基础。
- timing是关键:前10%训练内施加decay,在162M、405M、1.4B三个规模都有收益;同样长度窗口若从训练60%处开始,反而让最终损失变差。
- 162M调好的decay时刻表可直接迁移到1.4B:沿用未经重调的时刻表,训练73%处反超无decay基线。
- 作者建议:调会在训练中途关闭的正则化超参时,应在训练全程评估而非只看前期。
尚未确认
- 作者自曝局限:结论大多来自单种子;405M用AdamW+ReLU²时方法无效;405M与1.4B的下游评测未见相对恒定decay的明确优势;大模型上收益尚不明确;且尚不能解释为何这条step-size轨迹是好的。
为什么重要
- 这项工作把Muon下weight decay的收益归因到权重范数与等效步长的迁移机制,提示decay的时机可能比强度更重要,对Muon预训练的超参实践有直接参考价值;但泛化性与下游收益仍待更多种子与更大规模验证。
2026-11-19 ~ 2026-11-19 · 8 related posts
Primary sources
- New paper: weight decay in first 44% of Muon training hurts immediately but wins long-term — ziv_ravid · 2026-11-19
- Decay schedule tuned at 162M transfers to 1.4B, overtaking no-decay at 73% — ziv_ravid · 2026-11-19
- Timing matters: decay in first 10% of training helps, at 60% it hurts — ziv_ravid · 2026-11-19
- New paper: delayed effects of temporary weight decay in Muon pretraining — ziv_ravid · 2026-11-19
- [source] Weight norms never recover after early decay: 60-66% at training end — ziv_ravid · 2026-11-19
- [source] Temporary Weight Decay in Muon Pretraining Ends Worse but Finishes 0.028 nats Better — ziv_ravid · 2026-11-19
- [source] Norm-swap experiments trace Muon early-decay gains to weight norms — ziv_ravid · 2026-11-19
- Caveats: single-seed results, no clear win at 405M-1.4B evals — ziv_ravid · 2026-11-19