Kimi K3 adds MoonEP balancing and ViT pipeline parallelism to its training stack
nrehiew_ · x · 2026-07-29
The thread says Kimi K3 introduces MoonEP, an improvement on top of DeepEP, and focuses on systems work around MoE training.
- MoonEP needs perfect expert balancing, so redundant experts are drafted offline.
- The implementation is designed for zero-copy behavior and lower kernel-launch overhead.
- Memory savings come from a custom activation manager, a memory-efficient MoE setup, attention-residual kernels, ZeRO-2, and P2P allgathers for Muon.
- For the vision encoder, the system uses gather-KV-style context parallelism for long media tokens.
- The author also notes that the pipeline diagram includes PP, EP, ZeRO-2, and CP, and that it is the first time they have seen a ViT in a pipeline diagram.
Related event: Deep Dive into Kimi K3 Tech Report: Architecture and Training(18 posts)→
More from Infra
- Atlassian caps employee AI spend at up to $2,000 a month — nordicinst · 2026-07-29
- The Evolution of Enterprise AI Retrieval: Indexing Becomes Key to Accuracy and Latency — damianplayer · 2026-07-29
- Europractice MPW 2025 chart shows TSMC leads foundry designs with 251 — jwt0625 · 2026-07-29
- A practical Krea 2 LoRA guide targets 1024-res training on 16GB VRAM rigs — Endlesswoodtrail · 2026-07-29
- Kernel Forge uses MCTS to optimize CUDA kernels and beats PyTorch baselines on 14 cases — omarsar0 · 2026-07-29
- Users compare CPU/RAM offload setups for large models and mixed MoE workloads — Jorlen · 2026-07-29