Norm-swap experiments trace Muon early-decay gains to weight norms
ziv_ravid · x · 2026-11-19
Direct mechanism test at 162M: replaying the early-decay step-size trajectory with fixed norms and no decay yields a similar gain, and swapping norms between runs moves the gain with them (96% one way, 97% the other) — confirming the weight-norm/effective-step pathway.
Related event: Temporary Early Weight Decay in Muon Pretraining Shows Delayed Payoff(8 posts)→
More from Research
- Shanghai AI Lab's GameRealm: 120M frames of gameplay data for visual foundation models — Shanghai-AI-Laboratory · 2026-11-19
- KradleBench launches as a dynamic multi-agent benchmark pitting LLMs, world models and humans against each other — lmoroney · 2026-11-19
- KradleBench Launches: Minecraft Legend SethBling Dethroned by Model Astra 6.2 — fchollet · 2026-11-19
- Transformer Lab unveils Primus, a 'society of researchers' of AI agents, and is hiring — carsonfarmer · 2026-11-19
- New LessWrong Analysis: LLMs Seek the Grader's Score, Not the Actual Goal — 1a3orn · 2026-11-19
- tabgen-eval: an open-source eval framework and benchmark for synthetic tabular data — LucaAmb · 2026-11-19