Temporary Weight Decay in Muon Pretraining Ends Worse but Finishes 0.028 nats Better

ziv_ravid · x · 2026-11-19

A Meta team (first author Md Rifat Arefin) published an arXiv paper on the delayed effects of temporary weight decay in Muon-trained decoder-only Transformers (162M–3.4B params).

Key findings

Mechanism

Caveats: mostly single-seed; no help for AdamW+ReLU² at 405M; downstream evals show no clear win; the why remains open.

Original post →

More from Infra

Infra channel →