Muon optimizer eliminates need for batch size warmup
nrehiew_ · x · 2026-08-27
Loss curves reveal that using Muon alone results in very high gradient norms and residual activations. Findings indicate that architecture and optimizer changes shift optimal hyperparameters towards significantly larger batch sizes and learning rates. Key takeaways: batch size warmup is not needed with Muon due to large penalties at small sizes, and learning rates are more forgiving with a grace zone.
Related event: Qwen Details Training Recipes: Muon Optimizer and Residual Stream Findings(5 posts)→
More from Research
- Goodfire's Nick Cammarata: neural networks are "already clean" in a possible interpretability breakthrough — tszzl · 2026-10-15
- tszzl: interpretability breakthrough feels like my understanding of neural nets jumped five years — tszzl · 2026-10-15
- LightSuite: fast cell detection and whole-brain atlas registration on a standard workstation — hisspikeness · 2026-10-15
- Pathway CSO pitches BDH as post-Transformer architecture uniting memory, reasoning, adaptation — schwentker · 2026-10-15
- AI-designed molecular scissors cut Alzheimer's proteins — but is it really AI? — gerardsans · 2026-10-15
- EmbodiedEvo tops RoboDojo at 38.95% success by letting robots inherit failed practice — thetripathi58 · 2026-10-15