Timing matters: decay in first 10% of training helps, at 60% it hurts
ziv_ravid · x · 2026-11-19
Decay over the first 10% of training helps at 162M, 405M and 1.4B, while the same window starting at 60% makes final loss worse. Practical takeaway: when tuning a regularizer that switches off mid-run, compare runs at the end of the compute budget.
Related event: Temporary Early Weight Decay in Muon Pretraining Shows Delayed Payoff(8 posts)→
More from Research
- 'Physics Hasn't Progressed in 50 Years?' This New PRL Paper on Icy Friction Says Otherwise — CatAstro_Piyush · 2026-11-19
- TMol brings Rosetta-style all-atom molecular energetics into differentiable PyTorch — gottapatchemall · 2026-11-19
- ELLIS Tübingen safety team releases horizon-labs-agent-traces dataset on Hugging Face — NielsRogge · 2026-11-19
- Noam Brown confirms the math model claim; speculation points to an astra math fine-tune — haider1 · 2026-11-19
- NCME talk: generative psychometrics validates LLM-generated test items before human data — GolinoHudson · 2026-11-19
- Researchers say peer review has hit a breaking point as AI conference scale strains ICLR — LucaAmb · 2026-11-19