Muon optimizer makes the first MoE layer harder to load-balance, study finds
Marcos Hernanz · hf · 2026-11-04
A technical report from Universidad Autónoma de Madrid examines how training MoE language models with Muon (instead of AdamW) affects routing load balance, measuring the span of per-expert routing-bias values in models with 55M and 108M active parameters.
- The span grows when all tokens favor the same experts, and Muon increases the span in the first MoE layer, making its load balancing harder to maintain.
- Mitigations: centering router inputs or reducing the optimizer learning rate shrinks the imbalance.
More from Research
- Google and Mercor's PLE benchmark: all frontier agents score below 50% on 85 real engineering tasks — teortaxesTex · 2026-11-04
- Fable 5.1 hand-writes 100k-param classifier weights, beating backprop training — AccBalanced · 2026-11-04
- Cambridge's van der Schaar Lab opens PhD applications for 2027 entry — MihaelaVDS · 2026-11-04
- Tencent's Hyra 2.0 Generates Protein Sequences 57x Faster Than ProteinMPNN — YvesMulkers · 2026-11-04
- Researcher Urges Top AI Conferences to Desk-Reject AI-Generated Papers Amid Fake Data Scandals — Vegetable-Formal-753 · 2026-11-04
- Harvard Professors Admit AI Article Reviews Now Beat the Best Humans — iamfakhrealam · 2026-11-04