Kimi K3 pretraining combines PP, EP and ZeRO-1 with balanced MoE routing
SeunghyunSEO7 · x · 2026-07-28
Kimi K3’s pretraining stack mixes PP, EP, ZeRO-1 and context parallelism
The attached technical report page says Kimi K3 pretraining combines pipeline parallelism (with virtual stages), expert parallelism, ZeRO-1 data parallelism, pipeline ZeRO-2 gradient sharing, and context parallelism for 3T-class pretraining.
The paper section on MoonEP explains the motivation:
- Conventional expert-parallel MoE training suffers from imbalanced token loads across ranks.
- MoonEP aims for perfectly balanced expert-parallel training by using dynamic redundant experts.
- It plans a small number of redundant experts from router outputs, prefetches them before expert computation, and sends gradients back to the home ranks in backward.
- The report also emphasizes online planning with a near-optimal GPU kernel, zero-copy communication, and static shapes to keep overhead low.
The key claim is that the system keeps the overall computation flow of conventional MoE training while making load balance and communication much cleaner at scale.
Related event: Kimi K3 Tech Report: 2.8T MoE and Architectural Efficiency Breakthroughs(49 posts)→
More from Infra
- Bloomberg map shows Nvidia at the center of circular AI financing — SumitGup · 2026-07-28
- Moonshot open-sources MoonEP for balanced MoE expert parallelism — SeunghyunSEO7 · 2026-07-28
- POCKET: Running 35-Billion-Parameter LLMs Natively on iPhone — pmttyji · 2026-07-28
- Diffusers adds Nunchaku 4-bit diffusion inference with up to 50% less VRAM — dl_weekly · 2026-07-28
- SK hynix says AI demand is still “almost unbelievable” as memory boom stretches on — firstadopter · 2026-07-28
- OpenAI agent demo, open-weight momentum and AI’s carbon cost reshape enterprise thinking — jonerp · 2026-07-28