UltraEP: real-time load balancing for MoE training and inference
小红书技术REDtech · wechat · 2026-07-20
UltraEP brings real-time load balancing to MoE production
As MoE models scale toward trillion-parameter territory, expert parallelism has become the default approach for training and inference — but real workloads still suffer from severe imbalance across GPUs, causing idle compute, all-to-all bottlenecks, and even OOMs.
UltraEP, a system proposed by Xiaohongshu and Peking University, tackles this by using exact runtime routing signals to do real-time expert rebalancing per microbatch and per layer. Unlike predictive schemes such as EPLB that rely on stale historical traffic, UltraEP reacts to the actual gated load and dynamically copies hot experts within the high-bandwidth scale-up domain.
What it changes
- Control-plane optimization: jointly solves expert placement and token rerouting, with a quota-based search that keeps solve time under 100 µs at EP64.
- Data-plane optimization: uses persistent kernels, tiled transfers, and chunk-streaming relay to move weights/gradients efficiently and avoid communication hotspots.
- Memory efficiency: fixed slots for redundant experts and cross-layer buffer reuse reduce extra memory overhead dramatically.
Reported results
- Training: reaches 94.6% of ideal throughput on average, and is 42% faster than Megatron-LM.
- Inference prefill: reaches 90%–97% of ideal throughput and is 1.56× faster than SGLang.
- It also reduces rank imbalance from 1.30–4.01 down to 1.01–1.04.
The system is already deployed in production, including a full pretraining run of a 288B MoE model, where it sustained at least 92% of ideal performance over long training cycles. The team argues that as communication bandwidth improves, system-side load balancing will become a foundational layer for expert parallelism — and potentially for MoE RL workloads as well.
More from Companies & People
- Embodied AI startups need a different operational discipline, investor says — ___Mufasaa · 2026-10-27
- Reddit asks: How dependent are Chinese AI labs on the US ecosystem? — lol2funneeee · 2026-10-27
- Stanford HAI's Human-Centered AI for Social Impact program wraps third cohort — StanfordHAI · 2026-10-27
- Goodfire commits $10M to a 'Human Genome Project for AI' to reverse-engineer frontier models — CatAstro_Piyush · 2026-10-27
- Rumors: DeepSeek delayed on agents, losing its AI4Math edge as Kimi K3 breaks through — teortaxesTex · 2026-10-27
- SF 'Code Engineering' event picks up where 1,000-person harness engineering meetup left off — LangChain · 2026-10-27