New MoE paper: EQB exact global quantile balancing beats Kimi K3's approximate method
ChengleiSi · x · 2026-09-27
A new paper tackles MoE load balancing at two scales. EQB (Exact Quantile Balancing) computes exact global-batch BF16 quantiles with negligible communication, outperforming the approximate histogram variant used by Kimi K3. LEI (Load-Error Injection) injects local load errors into router-score gradients, beating the GShard auxiliary loss at comparable quality. Validated on 7.5B MoEs trained up to 500B tokens.
More from Research
- Google Research open-sources PaperOrchestra, a multi-agent framework for writing AI papers — google-research · 2026-11-16
- Kind ICLR Reviewer Raises Score Despite Unresolved Concerns, Citing the Paper's Value — MarkSchmidtUBC · 2026-11-16
- Hand-editing 0.2% of weights plus 100 CPU-minutes turns an English TTS model into an 18MB Sanskrit one — kastnerkyle · 2026-11-16
- Why chirality matters for AI for Science: enantiomers behave differently in biology — madeofAjala · 2026-11-16
- RSS 2027 submissions open with FAQ clarifying two-stage review and negative-result bar — GeorgiaChal · 2026-11-16
- Mobile AI Factory: agents auto-build on-device SLMs with 19x less GPU time — pmttyji · 2026-11-16