OasisKV: Scaling In-Decode KV Cache Beyond HBM with Lookahead Sparse Prefetching
Can Xiao, Sukmin Cho, Junbong We, Zhixiong Niu, Jianyi Cheng, Yiren Zhao, Youngjin Kwon, Yongqiang Xiong, Rui Ma, Junyi Liu
cs.DC
2026-08-08
OasisKV uses speculative-decoding draft tokens to predict one step ahead which KV blocks the next decode step needs, prefetching them into HBM asynchronously for 1.69x throughput at near-lossless accuracy.
Long-context and long-form reasoning workloads have turned LLM inference from compute-bound into memory-bound. The KV cache grows linearly with context: a 32B GQA model at 32.7K context (the measured average of production coding-agent traces) needs 8.6 GB of KV per request, so even dedicating all 80 GB of HBM to KV supports only nine concurrent requests. Scarce, expensive HBM capacity caps batch size and throughput directly.
The three existing routes each leave a gap. Sparse attention cuts KV reads but keeps the full KV in HBM, so capacity is unsolved. KV retrieval moves the full KV to CPU memory and stages only the active subset on the GPU, solving capacity but placing retrieval on the decode critical path and slowing every step. KV prefetch aims to overlap transfer with compute, but existing designs are pinned against the CPU-GPU IO bandwidth ceiling, neither reaching production throughput nor supporting disaggregated serving.
OasisKV's core observation is that decode-time attention is naturally sparse, and the KV blocks the next step needs can be predicted accurately one step ahead. The prediction signal is free: speculative decoding already drafts future tokens with an EAGLE-3 draft model, and those draft tokens are a training-free lookahead. Running a draft token forward over the current token's sparse KV, the predicted top-K set agrees with the true next-step query in 98.74% of blocks on average across layers.
Around that signal, the system does several things:
Accuracy (same 2048-token KV budget, each method read against its own full-attention anchor): on long-input LongBench v2, OasisKV drops only 0.4 on Qwen3-8B and 0.6 on Llama-3.1-8B; on long-output reasoning (AIME24/25, GPQA), pass@8 drops 0.66 and avg@8 0.35, the closest to full attention of any method and clearly better than Quest (down 2.6 to 3.5) and FreeKV.
Throughput is the headline:
| Setting | vs dense vLLM | Note |
| Qwen3-8B single-GPU 16K | 1398 vs 676 tok/s (2.1x) | concurrency 128 |
| Qwen3-8B AIME24 real reasoning | 2083 vs 1235 tok/s (1.69x) | 0.1 accuracy loss |
| Qwen3-235B TP8 | up to 1.9x | MoE, smaller KV share |
| PD disaggregation | 2.1 to 2.3x | remote partial fetch saves 2.2 to 2.6x decode-side host memory |
The 2048-token KV budget lifts 16K concurrency from 22 to 90-95. A key ablation states the bottleneck bluntly: loosening the fetch cap from 0.01 to fetch-all grows per-step traffic from 0.30 to 5.05 GB and collapses throughput from 2178 to 824 tok/s (a 2.6x drop), while accuracy wanders only between 74.9 and 77.4. Decode throughput is bound by PCIe bandwidth, not compute; the default 0.05 cap buys 2.5x the fetch-all throughput at 0.1 accuracy cost.
For serving teams running long-context, reasoning, or agentic workloads, OasisKV decouples KV capacity from HBM and trades it directly for batch size and throughput, reusing the draft tokens that speculative decoding already produces instead of training a dedicated predictor. It is implemented on vLLM, supports multi-GPU and disaggregated deployment, and is one of the few sparse-KV designs aimed squarely at production.
The authors concede that the draft token is currently used only as a lookahead and forced to reject, so its acceptance is not yet combined with speculative-decoding speedup (planned future work); on MoE models like Qwen3-235B the gain is smaller because the KV share is small and the per-step overhead makes TPOT higher than dense at low batch; and the prototype still retains a full KV per request in the prefill node's host memory until completion and does not yet support prefix caching (only an analytic TTFT model). One more from reading: the whole approach depends on having a ready EAGLE-3 draft head for each model; such heads are increasingly released as reusable artifacts, but obscure models still need one trained.