vLLM EPD upgrade cuts P90 TTFT 44-47% on multimodal document workloads
vllm_project · x · 2026-11-11
vLLM shipped major improvements to its EPD (encode-prefill-decode) disaggregation, letting multimodal workloads scale each stage independently instead of scaling the whole LM tier for more images.
Key numbers (DeepSeek V4.1 Flash multi-page document benchmark, concurrency 64-128):
- 44-47% lower P90 TTFT
- 6.6-9.5% higher token throughput per GPU, counting every encoder GPU
Engineering details from the new blog:
- Metadata-only requests avoiding duplicate preprocessing
- Mooncake/NIXL embedding transfers
- Cross-encoder caching
- Integration with vLLM Router, SMG, and llm-d for independent scaling of encoder and LM instances
DeepInfer's EPD disaggregated deployment also demonstrates flexible per-stage resource allocation.
Related event: vLLM's EPD Disaggregation Cuts Multimodal TTFT by Up to 47%(2 posts)→
More from Infra
- Baidu's ERNIE Evolve tops public bests on NanoGPT Speedrun, brings RSI to real inference optimization — xiaohu · 2026-11-11
- Baidu says RSI auto-optimized ERNIE inference, 6.24x decode speedup on Kunxin P800 — xiaohu · 2026-11-11
- Nvidia's supply commitments jump from $119B to $279B in one quarter, locking up 2027-28 memory — tengyanAI · 2026-11-11
- Train 27B models on a 24GB GPU: ternary optimizer cuts memory to 1 byte/weight — oguzeldereli · 2026-11-11
- NVIDIA expands Meta Muse partnership, invests $3B in Meta stock — armand_ruiz · 2026-11-11
- Did Opus 5.5 really cost $10B? Analysts argue final training run is under 10% of that — teortaxesTex · 2026-11-11