Kimi K3 uses FlashKDA to make chunkwise parallel context computation work
nrehiew_ · x · 2026-07-29
The thread explains the KDA chunkwise-parallel implementation used in Kimi K3.
- FlashKDA implements a chunkwise parallel form of KDA.
- Tensor parallelism is less useful here because KDA has fewer heads, so context parallelism is done across chunks.
- The key challenge is that the delta update rule depends on the previous state, which prevents independent computation.
- The solution splits the computation into two terms: one local delta-rule factor that can be computed independently, and another term that applies the delta rule after a single all-gather.
- The flow is summarized as: compute local delta-rule factor, compute local state, all-gather once, then apply the delta rule.
Related event: Deep Dive into Kimi K3 Tech Report: Architecture and Training(18 posts)→
More from Infra
- Atlassian caps employee AI spend at up to $2,000 a month — nordicinst · 2026-07-29
- The Evolution of Enterprise AI Retrieval: Indexing Becomes Key to Accuracy and Latency — damianplayer · 2026-07-29
- Europractice MPW 2025 chart shows TSMC leads foundry designs with 251 — jwt0625 · 2026-07-29
- A practical Krea 2 LoRA guide targets 1024-res training on 16GB VRAM rigs — Endlesswoodtrail · 2026-07-29
- Kernel Forge uses MCTS to optimize CUDA kernels and beats PyTorch baselines on 14 cases — omarsar0 · 2026-07-29
- Users compare CPU/RAM offload setups for large models and mixed MoE workloads — Jorlen · 2026-07-29