Kimi K3 swaps sign updates for quantile balancing across its 896 experts
Nouamanetazi · x · 2026-11-06
How do the newest MoEs balance expert load? Kimi K3, with 896 routed experts per layer, dropped the fixed-step bias update (loss-free sign update) for quantile balancing: each expert's bias is solved from a histogram of scores summed across GPUs. DeepSeek-V4.1-Flash keeps one bias per modality. The figure shows that on a repeated batch the sign update still leaves the busiest expert +25% above mean load, while quantile balancing converges to +8% with no step size to tune.
More from Research
- Making the planet's surface queryable by AI: Luke Davis unveils months-long project — dansitu · 2026-11-06
- Paul Dirac on where good ideas come from: relaxed Sunday walks, not active striving — CatAstro_Piyush · 2026-11-06
- GPT-6 Astra tops a 16-benchmark robot index, yet went 0/11 on real plug insertions — AIFlow_ML · 2026-11-06
- Probability reasons, statistics quantifies, ML predicts — Tivadar Danka untangles the three — TivadarDanka · 2026-11-06
- Burkov's True Positive Weekly #184: LLMs Can Natively Read & Write Neural Network Weights — burkov · 2026-11-06
- Six Claude agents lived 20 days in a simulated town and spontaneously built tools to cooperate — LiminalWanderings · 2026-11-06