Kimi K3 adds MoonEP balancing and ViT pipeline parallelism to its training stack
nrehiew_ · x · 2026-07-29
The thread says Kimi K3 introduces MoonEP, an improvement on top of DeepEP, and focuses on systems work around MoE training.
- MoonEP needs perfect expert balancing, so redundant experts are drafted offline.
- The implementation is designed for zero-copy behavior and lower kernel-launch overhead.
- Memory savings come from a custom activation manager, a memory-efficient MoE setup, attention-residual kernels, ZeRO-2, and P2P allgathers for Muon.
- For the vision encoder, the system uses gather-KV-style context parallelism for long media tokens.
- The author also notes that the pipeline diagram includes PP, EP, ZeRO-2, and CP, and that it is the first time they have seen a ViT in a pipeline diagram.
More from Infra
- Meta's open Muse Spark needs 493GB of memory — far beyond any consumer laptop — TejasKumar_ · 2026-10-16
- CoreWeave plunges 20% as credit markets price >60% default odds; is this dot-com all over again? — kimmonismus · 2026-10-16
- MiniCPM-V 4.7 35B-A3B hits 17+ tokens/s on iPad with ~3GB memory — techNmak · 2026-10-16
- China rolls out 'token factories' and a 2026-28 token economy action plan — nordicinst · 2026-10-16
- Vercel Sandbox Drives go GA: attach TBs of data in sub-second, fork instantly — cramforce · 2026-10-16
- NVIDIA launches RTX Spark for local AI on Windows PCs — ostrisai · 2026-10-16