vLLM's specialized decode path hits 829 P90 TPS per user on four B300 GPUs, 26% faster
vllm_project · x · 2026-10-21
Follow-up data from the vLLM team: beyond high throughput, its specialized decode path also excels at low latency — on real AgentX sessions it reaches 829 P90 TPS for a single user on four B300 GPUs, 26% faster than the standard path on GB300. Another post breaks down the $39/GPU/hr figure: at a P90 196 TPS per-user SLA, vLLM delivers 954K tokens/s per GPU counting cached input, input and output.
More from Infra
- vLLM says it can reproduce DeepSeek's inference performance, yielding $39/GPU/hr revenue — vllm_project · 2026-10-21
- vLLM matches DeepSeek's DeepInfer decode performance within 6%, generating $39/GPU/hr at off-peak pricing — vllm_project · 2026-10-21
- AMD brings EAGLE-4 speculative decoding to Instinct MI355X day 0, 1.6x faster than built-in MTP — hongyangzh · 2026-10-21
- TokenGate: learned token pruning speeds up LLM prefill up to 11.3x — Liana Patel · 2026-10-21
- Modal revives multiplayer game Curvytron to demo its new Sticky Sessions routing — charles_irl · 2026-10-21
- Seroter's reading list: TorchTPU open source, Claude Code's code review system — rseroter · 2026-10-21