KV Cache engineering: 1 GiB per 8K-token sequence, where LLM serving performance is won
techNmak · x · 2026-10-31
techNmak's deep dive on KV Cache engineering—the decisive battleground of LLM serving performance—paired with a detailed infographic.
Basics: during autoregressive generation, K/V for past tokens are cached and appended per layer; past queries aren't cached.
Memory math: raw full-attention KV ≈ 2 × layers × cached tokens × KV heads × head dim × bytes. With 32 layers, 8 KV heads, head dim 128, BF16: 128 KiB per position, 1 GiB for one 8,192-token sequence, 32 GiB for 32 unshared sequences—before weights, activations, and runtime overhead.
Levers and trade-offs:
- GQA/MQA shrink KV heads—an architectural choice, not a serving switch.
- PagedAttention manages KV in blocks (no compression) to avoid fragmentation and contiguous allocations.
- Prefix caching reuses KV for shared prefixes, cutting repeated prefill and TTFT; new-token generation still runs.
- KV quantization trades memory for quality/latency sensitivity; offloading to CPU introduces transfer bandwidth costs.
Production signals: watch KV usage with live sequence lengths, concurrency, free blocks, prefix hit rate, preemptions, queue time, TTFT, and ITL. KV shortages can cause queueing/preemption without a GPU OOM.
Bottom line: KV optimization is simultaneously a memory-allocation, bandwidth, scheduling, and cache-reuse problem—and one limit on GPU throughput at high concurrency.
More from Infra
- Baseten cuts 770GB RL weight sync to 15 seconds, 7.5x faster than prior SOTA — baseten · 2026-10-31
- Meta and SpaceX are already denting neocloud demand, expert says — bookwormengr · 2026-10-31
- Researchers encode 200+ properties of SGLang inference server, uncover 10 bugs — LingmingZhang · 2026-10-31
- Lux Capital hosts closed-door dinner on AI infra, agents and security with Cognition and Modal — graceisford · 2026-10-31
- GitHub incident hits Actions, Pages and Webhooks — damontal · 2026-10-31
- FastVideo ships install guide for NVIDIA DGX Spark, no aarch64 wheels needed — victormustar · 2026-10-31