vLLM-Omni ships KV reuse, FP8 and CUDA Graph optimizations with Qwen
vllm_project · x · 2026-09-20
The vLLM team details the engineering under the hood of vLLM-Omni, built in collaboration with Alibaba's Qwen team:
- Cross-step prefix KV reuse to avoid recomputing text and reference-image prefixes
- Dedicated CUDA Graphs to cut launch overhead for fixed-shape decode
- Request- and step-level continuous batching with phase-aware prefill/decode scheduling
- Tensor and Ulysses sequence parallelism
- Distributed VAE decoding with adaptive OOM recovery
- FP8 weights, FP8 prefix KV storage, and CPU offloading for different memory budgets
More from Infra
- Activist evicted in 2024 puts $5B Maryland AI data center on hold — ATTlKA · 2026-11-12
- Agent Substrate Calculator Estimates Session Resume Speed Under Load — pjausovec · 2026-11-12
- Dell expands AI Data Platform: agents have a data-context problem, not a model problem — DavidLinthicum · 2026-11-12
- SGLang Megakernel Pushes Muse Spark to 1,109 tok/s, 2.1x Over Baseline — hsu_byron · 2026-11-12
- Potion UI's local login system turns your PC into a multi-user AI server — cocktailpeanut · 2026-11-12
- AWS Finally Built Its Own AWS Emulator — super_linear · 2026-11-12