vLLM agentic serving details: packed KV saves 10% memory, Kimi K3 decode throughput up 2.7x
vllm_project · x · 2026-09-09
Parts 4-6 of vLLM's agentic serving thread detail the stack-level work: a packed KV layout for DeepSeek V4 saves 10% KV memory and reduces 92 tensors per block to 1; decode context parallelism delivers 2.7x decode throughput for Kimi K3 at the same TPOT, while prefill context parallelism runs DeepSeek V4's sparse MLA 2.65x faster than head sharding; simply capping prefill breaks head-of-line blocking for +93% TPGS and 2.3x better p90 interactivity. Real coding-agent traces show 43 median turns, 142K-token inputs vs. 444-token outputs, 96%+ prefix-cache hits, and 44% of sessions forking subagents.
More from coding & agent
- Anthropic's Claude Tag acts as on-call first responder for CI/CD failures — ClaudeDevs · 2026-09-09
- Inside Anthropic's Claude on-call agent that triages CI/CD failures in minutes — ClaudeDevs · 2026-09-09
- Agents now ship with a soul.md file, a nod to Peter Steinberger's influence — altryne · 2026-09-09
- Mitchell Hashimoto demos Superlogical remote persistent sessions, a full SSH replacement — iannuttall · 2026-09-09
- Catching LLM page overflow: a measure-and-repair loop for single-page LaTeX generation — Scholeristical · 2026-09-09
- OpenClaw ships 2026.9.3 with live browser automation, revocable share links, cloud repo work — heyneighbor · 2026-09-09