New MoE paper: EQB exact global quantile balancing beats Kimi K3's approximate method
ChengleiSi · x · 2026-09-27
A new paper tackles MoE load balancing at two scales. EQB (Exact Quantile Balancing) computes exact global-batch BF16 quantiles with negligible communication, outperforming the approximate histogram variant used by Kimi K3. LEI (Load-Error Injection) injects local load errors into router-score gradients, beating the GShard auxiliary loss at comparable quality. Validated on 7.5B MoEs trained up to 500B tokens.
More from Research
- Goodfire Says Reward Hacking May Soon Be a Thing of the Past With New Interp Research — burny_tech · 2026-09-27
- GPT-6 Astra reportedly proves weak Goldbach case unconditionally in 2 pages — yshan2u · 2026-09-27
- Open interpretability problem: convincingly editing a model's in-context beliefs — thebasepoint · 2026-09-27
- Decomposition-based interpretability isn't dead, but frontier models demand humility — thebasepoint · 2026-09-27
- Auto-research agents are coming, and peer review may shatter without defenses — askerlee · 2026-09-27
- Researcher argues most SAE work added little, backs Transluce's metamodel agenda — thebasepoint · 2026-09-27