Weight norms never recover after early decay: 60-66% at training end
ziv_ravid · x · 2026-11-19
At 1.4B, weight norms sit at 12-16% of the no-decay baseline when decay is removed and only recover to 60-66% by the end of training — the persistent norm suppression underpins the larger effective steps and final loss gains.
Related event: Temporary Early Weight Decay in Muon Pretraining Shows Delayed Payoff(8 posts)→
More from Research
- 'Physics Hasn't Progressed in 50 Years?' This New PRL Paper on Icy Friction Says Otherwise — CatAstro_Piyush · 2026-11-19
- TMol brings Rosetta-style all-atom molecular energetics into differentiable PyTorch — gottapatchemall · 2026-11-19
- ELLIS Tübingen safety team releases horizon-labs-agent-traces dataset on Hugging Face — NielsRogge · 2026-11-19
- Noam Brown confirms the math model claim; speculation points to an astra math fine-tune — haider1 · 2026-11-19
- NCME talk: generative psychometrics validates LLM-generated test items before human data — GolinoHudson · 2026-11-19
- Researchers say peer review has hit a breaking point as AI conference scale strains ICLR — LucaAmb · 2026-11-19