vLLM's agentic serving deep-dive: DeepSeek V4 Pro hits 83K tok/GPU-s at 106x lower cost than Opus 5
vllm_project · x · 2026-09-09
vLLM published a full-stack optimization deep-dive for agentic inference workloads, measured on SemiAnalysis's public AgentX benchmark.
What agent traffic looks like (from real coding-agent traces): median 43 turns per session; 142K-token median input vs. 444-token median output; 96%+ prefix-cache hit rate; 44% of sessions fork subagents — long prefixes, tiny outputs, constant reuse.
Key optimizations:
- KV cache: a packed KV layout for DeepSeek V4 saves 10% KV memory and collapses 92 tensors per block into 1.
- Parallelism must follow the model: decode context parallelism gives Kimi K3 2.7x decode throughput at the same TPOT; prefill context parallelism runs DeepSeek V4's sparse MLA 2.65x faster than head sharding.
- Scheduling: simply capping prefill breaks head-of-line blocking, yielding +93% TPGS and 2.3x better p90 interactivity.
- Hard-won lessons: pipeline parallelism only pays off on cold, long prompts; for short inter-turn workloads, session-sticky routing with warm KV cache can beat load balancing.
The economics: DeepSeek V4 Pro sustains 83K tokens per GPU-second under a strict p90 SLO of 50+ tok/s per user (130K at the frontier), while serving the same workload on Opus 5 at the same cache-hit rate costs 106x more — a measure of how much headroom open-weight models have when the stack is tuned for them.
More from coding & agent
- Anthropic's Claude Tag acts as on-call first responder for CI/CD failures — ClaudeDevs · 2026-09-09
- Agents now ship with a soul.md file, a nod to Peter Steinberger's influence — altryne · 2026-09-09
- How do you coordinate 10,000 AI agents on one problem? Hierarchical orchestration ideas emerge — pwlot · 2026-09-09
- Mitchell Hashimoto demos Superlogical remote persistent sessions, a full SSH replacement — iannuttall · 2026-09-09
- Catching LLM page overflow: a measure-and-repair loop for single-page LaTeX generation — Scholeristical · 2026-09-09
- OpenClaw ships 2026.9.3 with live browser automation, revocable share links, cloud repo work — heyneighbor · 2026-09-09