SparseEngine: Sparse-First Inference Engine
Jitai Hao, Quansheng Gu, Qiang Huang, Jun Yu
cs.LG
2026-09-30
SparseEngine hosts 15 sparse attention methods under one lifecycle contract, hitting about 10x vLLM decode throughput with KV eviction at large batches and 2.24x end-to-end speedup on agent traces.
Long-context agents pile up history with every tool call, and two GPU resources give out first: KV cache capacity and attention compute. Sparse attention is the standard fix, but it arrives in four families with incompatible needs. Dynamic selection (Quest, OmniKV) reads a slice of context while keeping the full cache; eviction (SnapKV, H2O) deletes entries outright; compression (Palu) and quantization (KIVI) change the payload itself.
Serving systems have not caught up. Vortex organizes everything around page operations, SPIN fixes a five-stage pipeline, Tangram specializes in non-uniform head-wise retention. A method entering someone else's framework needs rewriting, which is how the community accumulated per-model forks like LlamaSnapKV and Qwen3H2O. Figure 1 in the paper makes the gap concrete: no existing system combines broad method coverage with prefix caching and continuous batching.
SparseEngine moves the abstraction boundary. Instead of prescribing a workflow, it defines a lifecycle contract.
Two cross-request mechanisms sit on top. Chain Cache assigns each session a chainid and keeps compacted KV plus method state across turns; the logical token prefix stays intact, so a continuation prefills only the new suffix, and idle chains are reclaimed in LRU order. This fills a real gap: once eviction deletes physical KV, the radix prefix cache's completeness assumption breaks and the whole prefix had to be recomputed. Prefix-Cache Pruning lets an application name a history interval [L, R) and a keep ratio; a scoring policy such as KVzip picks the survivors, physical slots are released, and the logical prefix is preserved so later requests still match.
| Setting | Metric | Result |
| SnapKV, 128K inputs, largest tested batch | Aggregate decode throughput vs vanilla vLLM | 10x (both Qwen3-30B and GLM) |
| Quest / OmniKV, matched batch | Decode throughput vs vanilla vLLM | 1.5-2.6x |
| Quest, batch=2, 128K (Qwen3) | vs Vortex / HiSparse | 1.24x / 1.55x |
| SnapKV, same setting | vs Tangram | 1.55x |
| 20 aggregate comparisons (LongBench V1+V2) | Score gap vs original implementations | +0.17 points mean (variance 0.32) |
| SnapKV 8K + Chain Cache, Gasai replay | End-to-end speedup vs Vanilla | 2.24x (49.9 to 22.2 min) |
| SnapKV + Chain Cache, SWE-bench Lite | Resolved rate | GLM 24.7% (Vanilla 25.0%); Qwen3 8.3% (Vanilla 5.3%) |
Quality preservation is the strongest part: across 20 aggregate-score comparisons with the original method implementations, the mean gap is 0.17 points. AIME 2024 prices the speed: full attention reaches 81.7% accuracy, OmniKV holds 80.0% at 1.51x, SnapKV with a 4K budget drops to 58.3%, and StreamingLLM hits 3.36x with 18.3% accuracy. Bigger speedups cost more quality, and the paper shows the curve instead of hiding it. On tool-result pruning, immediate pruning at a 20% keep ratio costs a little accuracy; delaying it (lag=4, prune only the fifth-most-recent tool result, keep the latest four intact) restores 25.0% resolved, above the 23.7% unpruned baseline.
Teams serving long-context agents get one engine where 15 methods run under the same scheduler, prefix cache, and continuous batching, across 14 model variants including MLA (GLM-4.7-Flash) and hybrid linear attention (Qwen3.6), configurations where HiSparse and Tangram do not run at all. Read the 10x correctly: it comes from eviction freeing memory for larger batches; at matched concurrency the gain is 1.5-2.6x. For agent workloads the practical piece is Chain Cache, which lets eviction methods reuse compacted state across turns instead of recomputing the prefix every turn.
The authors' own list: quality and efficiency trade off; speedups shrink on shorter contexts (most methods land at 1.05-1.66x on AIME, where attention is a smaller fraction of runtime); LongBench V2 subtasks are small and sampled, so per-task scores fluctuate. Points that stand out on a close read: