KVMem: Virtualizing Million-Token Agent Workspaces on a Consumer GPU
Di Chai, Leye Wang, Zeshen Su, Zhiguo Xia, Zhihang Yu
cs.LG
2026-09-04
KVMem pages overflowed agent KV across GPU, RAM, and NVMe. A 24GB RTX 5090 laptop hosts a 1M-token workspace at ~50 tok/s; DeepSWE Pass@1 goes from 43.8% (compaction) to 48.4%.
A long-running coding agent fills its workspace with file reads, tool traces, edits, and logs. That history hits two walls. GPU memory has to hold weights, speculative-decoding state, runtime buffers, and KV at once, so the usable context on a consumer card is often far smaller than the advertised window. The model window itself is the second wall: Qwen3.6-27B is natively 256K, while agent traces already run to hundreds of thousands or millions of tokens.
Today's systems stay text-centric. They compact older history into a summary, then retrieve raw snippets when the summary is not enough. Compaction has to guess future relevance before later steps exist. A number in a tool dump or a user constraint can look disposable at compact time and become the whole task later. Retrieval can bring the text back, but the model already paid for those tokens once; stuffing them into the prompt means prefilling them again. Fidelity and compute are tied together.
KVMem treats the agent's workspace as a virtualizable resource. Addressable KV size is decoupled from what sits on the GPU, in the same spirit as OS virtual memory. Overflowed history lives as paged KV across GPU, host RAM, and NVMe. Each agent step materializes only a query-dependent working set into an execution view that fits the native window and the GPU KV budget.
Three mechanisms line up with when, what, and how.
Step-level scheduling. On eight OpenHands SWE-bench Lite rollouts, adjacent 128-token windows have 0.070 bits of attention KL inside a step and 2.59 bits across a step boundary, a 37.3x jump. KVMem therefore rebuilds the working set once per agent step, after prefill and before decode, and freezes it for the whole decode.
Query-conditioned retrieval. Default block size is 32 tokens. For every layer and KV head, the system stores a RoPE-stripped Mean-K vector and scores blocks in the serving model's own attention space. Historical attention is sparse: an analyzed window holds 103.1 blocks on average, yet the top-8 capture 66.5% of historical attention mass and the top-16 capture 77.0%. Sink and recent blocks are always kept; the rest of the budget is filled by score, then restored in chronological order.
Tiered movement. A repository view tracks where each block lives. An execution view holds only the selected blocks, packed into contiguous logical positions. RoPE-baked K is invalid after a remap, so an immutable position-independent raw K is kept off the active GPU cache and re-RoPEd at the new positions; V is copied as-is. Overlapping pages across consecutive steps stay on GPU. Frequently recalled blocks prefer host RAM over NVMe. Misses are packed into bulk transfers and pipelined across CPU gather, H2D copy, and GPU scatter plus re-RoPE.
The substrate is QW3, a custom C++/CUDA engine for Qwen. vLLM and SGLang restore KV along a monotonically growing prefix, which does not match selecting a sparse set and compacting it into new positions every step. The full Mean-K index stays in host memory; the GPU scores it in bounded tiles. When the backing store fills, the prototype falls back to text compaction, at a much coarser timescale than window overflow.
Controlled long-history tests use Qwen3.6-27B on the same QW3 backend, changing only the memory policy. Active-view budgets are 32K, 64K, or 100K depending on scale.
| Setting | Full Context | Compact-only | Compact+RAG | KVMem |
| LongMemEval-S acc / latency | 86.60% / 0.30s | 45.60% / 18.92s | 86.20% / 26.63s | 85.60% / 0.48s |
| MemoryAgentBench (>256K) score | n/a | 27.54 | 34.80 | 40.99 |
| AgentLongBench ≤256K success | 59.54% | 15.84% | 47.49% | 60.87% |
| AgentLongBench 1M success / latency | n/a | 32.00% / 380s | 42.00% / 416s | 50.00% / 0.73s |
Against Compact+RAG's post-compaction recovery latency, KVMem is 11.4-53.8x faster. At 512K AgentLongBench the two are essentially tied on success (53.0% vs 54.0%), with KVMem at 0.62s versus 264s. On in-window AgentLongBench KVMem even edges Full Context (60.87% vs 59.54%). The paper's reading is that dumping the entire history can hurt, and a query-selected subset can be cleaner.
On one 24GB RTX 5090 laptop, vLLM holds about 10K context and llama.cpp about 80K. KVMem matches llama.cpp's 80K execution view, then virtualizes the workspace to 1M, at about 50 tokens/s in a single session. On the server, with the execution view pinned at 64K, growing the workspace from 256K to 10M leaves GPU memory almost flat (33.9 to 34.9 GiB) while NVMe grows from 8.5 to 324 GiB and TTFT from 0.43s to 1.60s. The Mean-K index grows from 0.25 to 9.5 GiB, all on the host.
End-to-end agent runs switch to Qwen3.8-27B under the Claude Code harness, first sixteen DeepSWE v1.1 tasks in canonical order, four samples each.
| Config | Pass@1 | Pass@4 | Mean agent time | Prefill |
| Compaction-only | 43.8% | 81.3% | 52.5 min | 211.5 s |
| KVMem | 48.4% | 93.8% | 45.8 min | 95.0 s |
KVMem solves 15 of 16 tasks at least once; compaction solves 13. Decoded tokens drop from 154.4K to 125.6K.
Local long-running agents currently compact when the window fills, then pay prefill again to recover what the summary dropped. KVMem keeps the KV the model already computed. For someone running a Qwen-class coding agent on a laptop, the practical delta is an 80K executable view becoming a 1M addressable workspace on the same 24GB card, still at interactive decode speed.
This does not stretch a single attention window to a million tokens. Each step still sees a bounded execution view. What it saves is recomputing history that already had a KV. Against MemGPT or Mem0, the retrieval signal is the serving model's own attention space, and the payload is KV, not text. Against vLLM PagedAttention and LMCache, the hard part is a working set that changes every step and must be re-RoPEd into new compact positions.
You cannot drop this on a black-box API. The runtime has to own KV allocation, positional restore, and tier movement. The paper notes that a cloud provider could sell larger persistent workspaces and recoup cost from the gap between cached and fresh input tokens.
Virtualization is about addressable workspace size, not tokens jointly attended in one call. Reusing historical KV is not the same as prefilling the assembled view from text: re-RoPE fixes positions, V is copied, but the original causal context is gone. Retrieval can still miss a task-critical block. A larger execution view, when GPU budget allows, would lean less on the retriever.
Storage is the bill. A 10M-token workspace takes 324 GiB of NVMe and a 9.5 GiB host index. KV is far larger than text; that is the point of the space-for-compute trade. When the backing store fills, the prototype still falls back to text compaction. Multi-tenant serving, with several workspaces competing for GPU and disk, is unsolved.
DeepSWE covers the first sixteen canonical tasks, four seeds each, not the full leaderboard. Public model numbers in the same table use mini-swe-agent; the paired run uses Claude Code, so those rows are not a model bake-off. QW3 is a custom engine, and the vLLM versus llama.cpp capacity numbers also reflect different product goals: vLLM optimizes throughput, not single-session context size.