vLLM: agent sessions median 43 turns, 142K-token inputs vs 444-token outputs
vllm_project · x · 2026-09-09
vLLM profiled real coding-agent traffic: median 43 turns per session, 142K-token median input against only 444-token median output, 96%+ prefix-cache hit rate, and 44% of sessions forking subagents — long prefixes, tiny outputs, constant reuse. Companion data shows DeepSeek V4 Pro sustaining 83K tokens per GPU-second at a strict p90 interactivity SLO of 50+ tok/s per user (130K at the frontier tier), while serving the same workload on Opus 5 at the same cache-hit rate costs 106x more, highlighting the headroom open-weight models have with a tuned serving stack.
More from coding & agent
- Vovy launches a cursor-following voice agent for Mac, free tier gives 100 credits/mo — Scobleizer · 2026-10-27
- Redditor seeks real-world stacks for a persistent-memory personal AI that answers texts and emails — BrooksomidusGG · 2026-10-27
- Practical tip: use max effort for planning, xhigh for implementation in Claude Code — majidmanzarpour · 2026-10-27
- Chrome team explains WebMCP: agents get tools from your website, not a replacement for MCP — rseroter · 2026-10-27
- Agent-Native apps: why agents bypassing your UI can break business rules — rseroter · 2026-10-27
- 'My intelligence stack': developer splits AI coding work across Opus 5.5, Fable and Astra with human review gates — sh_reya · 2026-10-27