vLLM's hard-won lessons: pipeline parallelism falters on warm agent turns, 2.7x decode on Kimi K3
vllm_project · x · 2026-09-09
The vLLM project shares engineering lessons across three planes of the serving stack:
KV cache management: A packed KV layout for DeepSeek V4 saves 10% KV memory and cuts 92 tensors per block down to 1.
Parallelism follows the model: Decode context parallelism gives Kimi K3 2.7x decode throughput at the same TPOT; prefill context parallelism runs DeepSeek V4's sparse MLA 2.65x faster — but on DeepSeek V4, decode CP only matched DEP.
Scheduling: Pipeline parallelism is great on cold, long prompts, but on warm agent turns adding a few hundred tokens, bubbles eat the gain. Load balancing doesn't always beat simple session-sticky routing: for workloads with short inter-turn delays, preserving a warm KV cache can matter more than balancing the queue.
More from Infra
- DeepSeek v4 and GLM Now Run Faster Than vLLM and SGLang — jedisct1 · 2026-09-09
- Magic details >10x compute-efficient pretraining, eyes trillion-parameter models — Dr_Singularity · 2026-09-09
- Magic claims >10x more efficient pretraining, matches DeepSeek V4 Pro with 50x fewer FLOPs — Dr_Singularity · 2026-09-09
- Alexandr Wang backs claim that Scale's own compute lets it subsidize Muse's speed — alexandr_wang · 2026-09-09
- Epoch AI: GPT's Quadratic Latency vs Claude's Linear May Explain Pricing Gap — scaling01 · 2026-09-09
- What 100 GW of compute really means: 876 TWh a year and a country-scale power system — shyamalanadkat · 2026-09-09