HiSparse hybrid sparse attention lands in vLLM: 8x H200 concurrency jumps from 5 to 25 at 1M context
eliebakouch · x · 2026-09-09
- samsja19's team announced a collaboration with the vLLM team on sparse attention and HiSparse. Sparse attention picks only top-K tokens, cutting memory bandwidth pressure but not KV cache storage; in high-throughput wide-EP serving, long-sequence decode exhausts VRAM and prevents enough parallel requests to saturate compute.
- HiSparse offloads inactive KV to CPU, keeping an LRU hot cache on GPU. Since many K tokens are reused every decode step, the offloading is barely noticeable — big memory savings and higher concurrency, especially valuable for RL where throughput matters and you want to stay compute-bound.
- vLLM's Hybrid HiSparse builds on Sparse MLA (only top-K tokens attended, rest need not live on GPU). While there's room, KV stays on GPU; under pressure a request releases its coldest pages to host memory, keeps a small hot buffer for the indexer, and keeps decoding instead of being preempted.
- Benchmarks: one 8x H200 node, GLM 5.3, full 1M context, configured concurrency 32 — plain KV offloading kept 5-6 requests running, Hybrid HiSparse kept 19-25. Hot pages are ordinary KV blocks from the same pool (Hybrid Memory Allocator), with a single fused kernel.
Related event: vLLM Adds Hybrid HiSparse, Quadrupling Long-Context Concurrency(2 posts)→
More from Infra
- Podcast Dives Into Broadcom Custom ASICs, 2027 Supply Bottleneck, and Nvidia's Hugging Face Deal — BenBajarin · 2026-09-09
- Magnitude open-sources Apple silicon inference server that auto-tunes local models for your Mac — nickbaumann_ · 2026-09-09
- Cerebras paper: layer dropout saves up to 25% training FLOPs and yields 1.55x faster decoding — burny_tech · 2026-09-09
- Viettel unifies GPU fleet into Token-as-a-Service platform with three open source layers — PyTorch · 2026-09-09
- Put per-turn action schemas in the last user message to preserve prompt caching — Low_Bad_6585 · 2026-09-09
- minnow: An Open-Source Fast Inference Server for LLaDA2.2 Diffusion LMs — coder543 · 2026-09-09