vLLM P/D prototype hits 91.24% prefix cache hit rate on 12× GB300
vllm_project · x · 2026-10-15
The vLLM project shared pre-release benchmarks for its P/D (prefill/decode disaggregation) prototype running DEP4 prefill connected to DEP8 decode across 12× GB300, using FP8, NIXL transfer, and a 384 GiB MooncakeStore cache.
Key results:
- 514,784 prefix-hit tokens recorded, a 91.24% prefix reuse rate
- Two 1,000-request OCR runs scored 90.4% and 89.4% with zero request errors
- Path: Prefill (TP1·DEP4) → NIXL transfer of sparse state/KV → Decode (TP1·DEP8), with the cache attached to the transfer layer
The team stresses this is a pre-release, pre-rebase test from an open PR; final-release validation is separate and still pending.
Related event: vLLM Shares P/D Prototype and MTP3 Benchmark Results(2 posts)→
More from Infra
- Inworld opens up its consumer-AI inference API, citing tens of trillions of tokens a month — thetripathi58 · 2026-10-15
- When will 16GB machines run local LLMs on par with Opus-class models? — bit_freak · 2026-10-15
- vLLM ships Day-0 support for Step 5 Preview, cuts sparse prefill latency 2.2% on B300 — vllm_project · 2026-10-15
- vLLM details Step 5 Preview's region-based sparse attention: score summaries, then attend — vllm_project · 2026-10-15
- vLLM announces Day-0 support for Step 5 Preview across Blackwell and Hopper — vllm_project · 2026-10-15
- UW's COUNT stores n-gram counts on host memory, giving 1B LMs the equivalent of 57% more compute — UW · 2026-10-15