vLLM-Omni ships KV reuse, FP8 and CUDA Graph optimizations with Qwen
vllm_project · x · 2026-09-20
The vLLM team details the engineering under the hood of vLLM-Omni, built in collaboration with Alibaba's Qwen team:
- Cross-step prefix KV reuse to avoid recomputing text and reference-image prefixes
- Dedicated CUDA Graphs to cut launch overhead for fixed-shape decode
- Request- and step-level continuous batching with phase-aware prefill/decode scheduling
- Tensor and Ulysses sequence parallelism
- Distributed VAE decoding with adaptive OOM recovery
- FP8 weights, FP8 prefix KV storage, and CPU offloading for different memory budgets
More from Infra
- SpaceX's orbital AI data centers weigh up to 4,000 kg each, filing seeks 1M satellites — XFreeze · 2026-09-20
- Local Models for Personal Agents: GPT Luna Surprises a Coding-Agent Veteran — gized00 · 2026-09-20
- OpenAI hardware VP details first custom chip Jalapeño and its nine-month tape-out — bigdata · 2026-09-20
- focus-llama: a llama.cpp fork implementing Declarative Attention for up to 0.71x decode time — Ok-Shower7286 · 2026-09-20
- VTrain, a Vulkan-based resident trainer, fixes memory leak and offloads more work to GPU — Savantskie1 · 2026-09-20
- vLLM ships day-0 support for Qwen-Image-2.1 with cross-step prefix KV cache — Alibaba_Qwen · 2026-09-20