MoonEP keeps MoE training perfectly balanced with dynamic redundant experts
stochasticchasm · x · 2026-07-28
This post shares a paper section on Perfectly Balanced Expert-Parallel MoE Training and the MoonEP approach.
- Conventional expert-parallel schemes suffer from token-load imbalance across ranks, which hurts throughput and causes memory fragmentation because routed-expert activations change shape dynamically.
- MoonEP aims to keep the overall compute flow of standard EP while adding online planning and migration of redundant experts.
- In the forward pass, it plans redundant experts from router outputs of the current micro-batch and layer, then prefetches them before routed-expert computation.
- In the backward pass, gradients are staged in a local reduce buffer and later reduced back to the home ranks.
- The paper argues that each rank must receive exactly S × K tokens and proves a balanced plan exists with at most E / R redundant experts per rank, which is presented as a tight bound.
- Prior methods like ECHO and UltraEP either preset redundant experts or cap per-rank tokens, which can still leave imbalance or force training to stop when no feasible plan exists.
More from Infra
- SkyPilot says serving Kimi K3 needs multi-node inference and a full stack — skypilot_org · 2026-07-28
- Ilya Sutskever’s SSI partners with Nvidia in a deal Bloomberg pegs at $5 billion — RebeccaBellan · 2026-07-28
- Moonshot’s Mooncake serving stack boosts long-context throughput by up to 525% — stochasticchasm · 2026-07-28
- Fireworks says Kimi K3 matches Opus 5 quality at 2x–4.6x lower task cost — lqiao · 2026-07-28
- Muon orthogonalization goes peer-to-peer to cut all-gather overhead — stochasticchasm · 2026-07-28
- MoE training paper proposes workload-aware expert-GEMM scheduling — stochasticchasm · 2026-07-28