Decay schedule tuned at 162M transfers to 1.4B, overtaking no-decay at 73%
ziv_ravid · x · 2026-11-19
A schedule picked on the 162M model transfers to 1.4B without retuning: it passes the no-decay run at 73% of training and ends ahead of the best constant decay tried. Setup: decoder-only Transformers, 162M to 3.4B, trained with Muon, weight decay on for an early window then off.
Related event: Temporary Early Weight Decay in Muon Pretraining Shows Delayed Payoff(8 posts)→
More from Research
- Shanghai AI Lab's GameRealm: 120M frames of gameplay data for visual foundation models — Shanghai-AI-Laboratory · 2026-11-19
- KradleBench launches as a dynamic multi-agent benchmark pitting LLMs, world models and humans against each other — lmoroney · 2026-11-19
- KradleBench Launches: Minecraft Legend SethBling Dethroned by Model Astra 6.2 — fchollet · 2026-11-19
- Transformer Lab unveils Primus, a 'society of researchers' of AI agents, and is hiring — carsonfarmer · 2026-11-19
- New LessWrong Analysis: LLMs Seek the Grader's Score, Not the Actual Goal — 1a3orn · 2026-11-19
- tabgen-eval: an open-source eval framework and benchmark for synthetic tabular data — LucaAmb · 2026-11-19