vLLM adds bi-directional KV transfer, improving multi-turn agent TTFT up to 3.3x
AWS ML Blog · rss · 2026-10-20
AWS ML Blog details bi-directional KV cache transfer contributed to vLLM, targeting redundant prefill work in multi-turn conversational and agentic workloads.
- Problem: one-directional (prefill→decode) KV flow forces prefill to recompute KV for previously generated tokens each turn, and full cache misses mean recomputing the entire conversation history
- Design: extended NIXL connector for D→P reads; decode retains eligible KV blocks under a fixed TTL (default 480s); a KV-cache-aware router caches block-location metadata keyed by conversationid; kvrecomputethreshold (default 64 tokens) trades off reuse vs local recompute
- Results: on EFA-connected P5en 1P1D deployments, TTFT improved up to 1.6x for Qwen3-32B and 3.3x for GLM-5-FP8 in multi-turn benchmarks
- Validated on Trainium first, contributed upstream as an accelerator-agnostic vLLM capability
More from Infra
- Cadence announces sovereign AI cloud for European chip designers — karlfreund · 2026-10-23
- SGLang Deep-Dive Blog Series Breaks Down KV Cache, Posts up to 15.5x Throughput Gains — BanghuaZ · 2026-10-23
- DynaLM: Open-Source CPU-First LLM Inference Engine Seeks Feedback — Unlikely_Safety_7456 · 2026-10-23
- Capital Is Now the AI Buildout Bottleneck, Says Analyst Citing Jensen Huang — JOBhakdi · 2026-10-23
- Cerebras explains how it deploys trillion-parameter models on wafer-scale chips — bookwormengr · 2026-10-23
- IFM open-sources RL360, the agentic RL infra behind K2 Horizon's training — rupspace · 2026-10-23