Kimi K3 Infrastructure: FlashKDA and Chunkwise Parallel Optimization
nrehiew_ · x · 2026-07-29
The author dives into Kimi K3's infrastructure, specifically FlashKDA. Since TP is inefficient for KDA due to fewer heads and unspllicable recurrence, context parallelism is done across chunks.
To solve the state dependency in the delta update rule, they break the computation into independent delta rule factors and local states, requiring only a single All Gather to apply the delta rule, significantly boosting efficiency.
More from Infra
- Reflection AI open-sources Beam under Apache 2.0 with 40+ ecosystem partners — zhyncs42 · 2026-10-16
- NVIDIA launches RTX Spark for local AI on Windows PCs — ostrisai · 2026-10-16
- SkyRL ships Day-0 RL support for Beam with +62% LoRA decode throughput in vLLM — vllm_project · 2026-10-16
- DFlash2 delivers up to 3.03x decode-only speedup for Beam across five benchmarks — vllm_project · 2026-10-16
- vLLM's Beam serving kernels cut GB200 forward time from 8.15 to 6.47 ms at TP8 batch 128 — vllm_project · 2026-10-16
- Reflection's 501B MoE Beam hits 463 tok/s on 8x GB200 with NVFP4 at day 0 — zhyncs42 · 2026-10-16