ID Balancing applies PID control to stabilize MoE training at 256x sparsity
teortaxesTex · x · 2026-10-01
A new arXiv paper unifies auxiliary-loss-free MoE load balancing methods as incomplete PID controllers: DeepSeek's loss-free method acts as a fixed-step integral controller and Kimi K3's Quantile Balancing as a generalized proportional controller.
Building on this control-theory view, the authors propose ID Balancing, an Integral-Derivative controller that scales its integral term with load error and activates the derivative term only when imbalance worsens.
Across Top-10, Top-5, and Top-3 routing over 768 experts, it cuts worst-case backbone MaxVio by over 50% and training-average MinVio by 12% versus the best baselines in the Top-3 setting, supporting scaling from 18.9B to 69.9B total parameters at extreme sparsity.
More from Research
- Technion's MIST stress test finds irrelevant images shift 20% of VLM judge labels regardless of content — Technion · 2026-10-01
- CISPA study shows latent multi-agent communication channels can push harmful compliance from 27.9 to 76.9 — cispa · 2026-10-01
- Google Research: generative UI lets teachers build learning simulations, rated 8/10 — dl_weekly · 2026-10-01
- A 1-cent verifier catches 61% of AI agents falsely claiming task completion, paper finds — alex_verem · 2026-10-01
- How Tri Dao's FlashAttention became a cornerstone of modern LLM training — thisdudelikesAI · 2026-10-01
- Looped Transformers: when is recurrent depth worth the extra compute? — ArchitectingAI · 2026-10-01