vLLM agentic serving details: packed KV saves 10% memory, Kimi K3 decode throughput up 2.7x
vllm_project · x · 2026-09-09
Parts 4-6 of vLLM's agentic serving thread detail the stack-level work: a packed KV layout for DeepSeek V4 saves 10% KV memory and reduces 92 tensors per block to 1; decode context parallelism delivers 2.7x decode throughput for Kimi K3 at the same TPOT, while prefill context parallelism runs DeepSeek V4's sparse MLA 2.65x faster than head sharding; simply capping prefill breaks head-of-line blocking for +93% TPGS and 2.3x better p90 interactivity. Real coding-agent traces show 43 median turns, 142K-token inputs vs. 444-token outputs, 96%+ prefix-cache hits, and 44% of sessions forking subagents.
More from coding & agent
- Vovy launches a cursor-following voice agent for Mac, free tier gives 100 credits/mo — Scobleizer · 2026-10-27
- Redditor seeks real-world stacks for a persistent-memory personal AI that answers texts and emails — BrooksomidusGG · 2026-10-27
- Practical tip: use max effort for planning, xhigh for implementation in Claude Code — majidmanzarpour · 2026-10-27
- Chrome team explains WebMCP: agents get tools from your website, not a replacement for MCP — rseroter · 2026-10-27
- Agent-Native apps: why agents bypassing your UI can break business rules — rseroter · 2026-10-27
- 'My intelligence stack': developer splits AI coding work across Opus 5.5, Fable and Astra with human review gates — sh_reya · 2026-10-27