Push-based KV transfer lands in vLLM, cutting TTFT by up to 3x for disaggregated inference
AWS ML Blog · rss · 2026-10-20
AWS ML Blog introduces a push-based NIXL connector contributed to vLLM, improving KV transfer for prefill-decode disaggregated inference.
- Pull mode serializes router relay, decode-side block allocation, and NIXL READ on the TTFT critical path; push mode overlaps them
- Design: decode registers destination blocks at allocation time; prefill-side background threads handle connection setup, descriptor prep, and NIXL WRITE; rendezvous handles out-of-order arrivals and reissued requests; differing block layouts supported on each side
- Results: on EFA-connected P5en 1P1D deployments, up to 3x better TTFT and 30% better TPOT for Qwen3-32B vs pull mode
- Built on Trainium clusters first, contributed upstream as an accelerator-agnostic vLLM feature
More from Infra
- Bloom Energy Q3 revenue hits $1.24B, up 138% YoY, raises FY26 guidance to $4.4-4.6B — Beth_Kindig · 2026-10-23
- 1M+ PHP services patched in 24h, but ~40M still exposed to PHP2Shell CVEs — FanaHOVA · 2026-10-23
- Trader dismisses AI circular financing theory: every new megawatt booked at rising spot prices — JOBhakdi · 2026-10-23
- Rumor: New Architectures Could Cut Cerebras K3 Wafer Needs From 60 to 10 — bookwormengr · 2026-10-23
- Nvidia plunges 7.21% to $212.31 in a day, still holding a $5.2T market cap — michalmalewicz · 2026-10-23
- Serverless Is a Billing Model: How to Price Stateful Compute by Usage — s4chinraja · 2026-10-23