Kimi K3 Inference Architecture and Cache Challenges

青稞AI · wechat · 2026-07-19

This deep dive breaks down the deployment challenges of Kimi K3 from an inference and systems perspective, focusing heavily on the KDA + AttnRes hybrid attention mechanism, Prefix Cache, KV Cache management, and memory allocation strategies under mixed states.

The author argues that while K3's architecture is computationally suited for linear attention, it introduces new engineering hurdles for inference stacks like vLLM / openinfer:

P/D Disaggregation: Linear attention significantly reduces the data transferred between prefill and decode, theoretically allowing cheaper or even heterogeneous hardware to handle prefill.
Prefix Cache: Compared to a "snapshot strategy," the author discusses how to determine snapshot frequency within limited space, index linear attention states alongside standard KV Cache, and notes that shared prefixes and workloads impact cache hit rates.
Memory Allocation: Because the memory footprint ratio between linear states and KV Cache dynamically shifts, it cannot be simply partitioned at a fixed ratio. The author suggests an approach similar to JENGA, using larger page granularity to accommodate both states and reduce fragmentation.
Decode Efficiency: While linear attention reduces per-request HBM usage and could significantly boost batch sizes, it also amplifies the pressure on all-to-all communication and group gemm.

The article concludes with a practical limitation: the author's own openinfer does not yet support prefix caching for hybrid attention, meaning exploration is currently limited to performance and scheduling optimizations.

Original post →

Kimi K3 Inference Architecture and Cache Challenges

More from Infra