kvcached brings virtual memory to LLM KV cache, deployed on 10K+ GPUs
techNmak · x · 2026-09-25
The idea: kvcached applies one of the oldest OS concepts—virtual memory—to the biggest bottleneck in LLM serving: KV cache GPU memory allocation.
The problem: vLLM and SGLang can reserve large KV-cache pools per model, and PagedAttention manages blocks efficiently within a model, but physical GPU memory is still hard to redistribute across model instances.
The approach: kvcached decouples the virtual KV address space from underlying physical GPU memory. A model reserves a large virtual space while physical pages are mapped only on demand, then reclaimed and reused by other models.
Traction: the underlying Prism system was published at OSDI '26, and the kvcached balloon driver has been deployed across 10K+ GPUs.
More from Infra
- Former Intel CEO calls HBM "lousy" at Hot Chips 2026 as High Bandwidth Flash looms — Glittering_Depth_722 · 2026-09-25
- MLPerf Training v6.1 adds first LLM post-training benchmark: agentic RL on a 397B model — TheKanter · 2026-09-25
- IEEE plenary talk: micro-optimizations across the full stack, from silicon to models — fooobar · 2026-09-25
- Dev builds local AI GTM workflow, argues the next platform entry point is hardware-bound — dotey · 2026-09-25
- US Faces Memory Chip Conundrum as AI-Critical Prices Skyrocket, WSJ Reports — pstAsiatech · 2026-09-25
- Qwen Flash Next IQ4_XS beats 27B FP8 on MMLU-Pro, GPQA and GSM8K in community eval — smallDeltaBigEffect · 2026-09-25