UltraEP brings real-time MoE balancing to Xiaohongshu’s rack-scale inference stack
机器之心 · wechat · 2026-07-29
This long article explains UltraEP, a real-time MoE load-balancing system developed by Xiaohongshu’s dotsinfra team.
What problem it targets
- In rack-scale MoE training and inference, expert load can swing wildly across GPUs.
- Static or history-based balancing leaves hot experts overloaded and can cut real throughput far below ideal performance.
What UltraEP does
- It reads the actual token load after the gate and before token dispatch.
- It jointly solves expert replication and token rerouting per microbatch and per layer.
- It uses reserved spare slots to place temporary expert copies without expensive memory churn.
- Replicas and gradients are shared across MoE layers to keep memory overhead low.
Reported results
- Extra overhead on the forward path is around 300 μs.
- On Qwen3-235B-A22B, the total critical-path overhead stays within that range.
- Across 106B–671B MoE models, throughput reaches 94.3% of ideal force-balanced performance.
- Compared with Megatron-LM / SGLang, average throughput improves by 1.49×.
- The most overloaded rank is reduced from 4.01× average load to 1.04×.
Why it matters
The piece frames UltraEP as a production-ready runtime layer for MoE systems: it complements token-communication stacks such as DeepEP by handling the separate problem of live expert scheduling after routing decisions are known.
More from Infra
- SK hynix reportedly signs long-term contracts with about 10 major customers — dejavucoder · 2026-07-29
- Shanghai Aishengna is said to be manufacturing DUV lithography tools — zephyr_z9 · 2026-07-29
- A vendor-agnostic Vulkan backend cuts edge inference latency from 30 ms to 3 ms — ppchaos · 2026-07-29
- pdf-mcp turns technical PDFs into structured text, images, and searchable context — tom_doerr · 2026-07-29
- Moonshot’s Kimi K3 is a 2.8T open-weight MoE model with 1M-token context — alex_verem · 2026-07-29
- New scaling law paper says repetition can beat paraphrasing for some pretraining regimes — burny_tech · 2026-07-29