HiSparse hybrid sparse attention lands in vLLM: 8x H200 concurrency jumps from 5 to 25 at 1M context

eliebakouch · x · 2026-09-09

samsja19's team announced a collaboration with the vLLM team on sparse attention and HiSparse. Sparse attention picks only top-K tokens, cutting memory bandwidth pressure but not KV cache storage; in high-throughput wide-EP serving, long-sequence decode exhausts VRAM and prevents enough parallel requests to saturate compute.
HiSparse offloads inactive KV to CPU, keeping an LRU hot cache on GPU. Since many K tokens are reused every decode step, the offloading is barely noticeable — big memory savings and higher concurrency, especially valuable for RL where throughput matters and you want to stay compute-bound.
vLLM's Hybrid HiSparse builds on Sparse MLA (only top-K tokens attended, rest need not live on GPU). While there's room, KV stays on GPU; under pressure a request releases its coldest pages to host memory, keeps a small hot buffer for the indexer, and keeps decoding instead of being preempted.
Benchmarks: one 8x H200 node, GLM 5.3, full 1M context, configured concurrency 32 — plain KV offloading kept 5-6 requests running, Hybrid HiSparse kept 19-25. Hot pages are ordinary KV blocks from the same pool (Hybrid Memory Allocator), with a single fused kernel.

More from Infra