UltraEP: real-time load balancing for MoE training and inference

小红书技术REDtech · wechat · 2026-07-20

UltraEP brings real-time load balancing to MoE production

As MoE models scale toward trillion-parameter territory, expert parallelism has become the default approach for training and inference — but real workloads still suffer from severe imbalance across GPUs, causing idle compute, all-to-all bottlenecks, and even OOMs.

UltraEP, a system proposed by Xiaohongshu and Peking University, tackles this by using exact runtime routing signals to do real-time expert rebalancing per microbatch and per layer. Unlike predictive schemes such as EPLB that rely on stale historical traffic, UltraEP reacts to the actual gated load and dynamically copies hot experts within the high-bandwidth scale-up domain.

What it changes

Control-plane optimization: jointly solves expert placement and token rerouting, with a quota-based search that keeps solve time under 100 µs at EP64.
Data-plane optimization: uses persistent kernels, tiled transfers, and chunk-streaming relay to move weights/gradients efficiently and avoid communication hotspots.
Memory efficiency: fixed slots for redundant experts and cross-layer buffer reuse reduce extra memory overhead dramatically.

Reported results

Training: reaches 94.6% of ideal throughput on average, and is 42% faster than Megatron-LM.
Inference prefill: reaches 90%–97% of ideal throughput and is 1.56× faster than SGLang.
It also reduces rank imbalance from 1.30–4.01 down to 1.01–1.04.

The system is already deployed in production, including a full pretraining run of a 288B MoE model, where it sustained at least 92% of ideal performance over long training cycles. The team argues that as communication bandwidth improves, system-side load balancing will become a foundational layer for expert parallelism — and potentially for MoE RL workloads as well.

Original post →

UltraEP: real-time load balancing for MoE training and inference

More from Companies & People