vLLM's Hybrid HiSparse keeps decoding past HBM limits: 19-25 vs 5-6 concurrent 1M-context requests
vllm_project · x · 2026-09-09
The vLLM project (built by RedHat AI, Prime Intellect and community) introduced Hybrid HiSparse: since Sparse MLA only attends to top-K tokens, KV no longer needs to fully reside on GPU, so a request keeps decoding after its KV outgrows HBM instead of being preempted.
Under memory pressure, coldest KV pages are offloaded to host memory while a small hot buffer serves the indexer. Key details: hot pages are ordinary KV blocks from the same pool (Hybrid Memory Allocator); a single fused kernel handles resident, hot and missing rows and is CUDA-graph capturable; prefix caching, OffloadingConnector, P/D imports and MTP all keep working.
Demo: one 8× H200 node, GLM 5.3, full 1M context, concurrency 32 — KV offloading kept 5-6 requests running vs 19-25 with Hybrid HiSparse.
More from Infra
- Inference is turning GPU compute into a tradable commodity — ArtificialAnlys · 2026-09-09
- Cohere open-sources megakernel serving engine, up to 1.58x faster than vLLM — cohere · 2026-09-09
- Solving Navier-Stokes cost 130B output tokens — up to $18M depending on model pricing — mitsuhiko · 2026-09-09
- Baseten cuts delta weight syncs for frontier models to under 40 seconds — baseten · 2026-09-09
- Dell says DRAM, NAND shortages persist and nearly all leading-node products are constrained — Beth_Kindig · 2026-09-09
- Alphabet's CapitalG backs AI chip startup Celero at $3 billion valuation — dinabass · 2026-09-09