vLLM ships Day-0 support for Step 5 Preview, cuts sparse prefill latency 2.2% on B300
vllm_project · x · 2026-10-15
- vLLM announced Day-0 support for Step 5 Preview, covering sparse attention, MoE, multi-token prediction and vision, with serving work across Blackwell and Hopper GPUs.
- Architecture: sparse attention starts from region summaries — an indexer scores summaries, selects regions, and attention runs only on selected tokens, with unfinished regions kept in the local window; the backbone combines sliding-window and sparse attention with MoE and MTP alongside a vision path.
- Kernel results (B300, DEP8, MTP3, FP8 KV, 60K-token input, MegaMoE path): indexer logits kernel latency down 43%, top-k kernel down 84%, overall prefill down 2.2%; the work also removes 7 runtime JIT warmups and caps chunks at 4,096 tokens.
Related event: vLLM Announces Day-0 Support for StepFun Step-5 Preview(2 posts)→
More from Infra
- Inworld opens up its consumer-AI inference API, citing tens of trillions of tokens a month — thetripathi58 · 2026-10-15
- When will 16GB machines run local LLMs on par with Opus-class models? — bit_freak · 2026-10-15
- vLLM P/D prototype hits 91.24% prefix cache hit rate on 12× GB300 — vllm_project · 2026-10-15
- vLLM details Step 5 Preview's region-based sparse attention: score summaries, then attend — vllm_project · 2026-10-15
- vLLM announces Day-0 support for Step 5 Preview across Blackwell and Hopper — vllm_project · 2026-10-15
- UW's COUNT stores n-gram counts on host memory, giving 1B LMs the equivalent of 57% more compute — UW · 2026-10-15