PagedAttention explained: managing KV cache like OS virtual memory
NielsRogge · x · 2026-11-10
A Papers with Code explainer breaks down PagedAttention, the idea behind vLLM: treat the KV cache like OS virtual memory.
- Each sequence's cache is split into fixed-size blocks that can live anywhere in GPU memory, instead of one contiguous buffer reserved for max length
- Benefits: less memory waste, no external fragmentation, internal fragmentation limited to the last block, on-demand allocation, and block sharing via copy-on-write for parallel samples, beam candidates and shared prefixes
- vLLM built on it improved throughput 2–4x at equal latency vs FasterTransformer and Orca; the paging idea is now standard in LLM inference engines
- 210 papers on Papers with Code use it
More from Infra
- Seroter's daily reading list: new TPUs, Claude 5.2 price cut, and the future of the PM role — rseroter · 2026-11-10
- DOE Funds $11.5M GridFM 2.0 AI Project Targeting 1B Grid Scenarios in 24 Hours — ingliguori · 2026-11-10
- NVIDIA open-sources Molt: a 9k-line RL framework where the whole setup is one Python file — algo_diver · 2026-11-10
- Microsoft Foundry adds Kling, LTX, and MiniMax video models via new fal partnership — gorkem · 2026-11-10
- Uno paper: diffusion adapters on AR LLMs beat speculative decoding with lossless speedups — LucaAmb · 2026-11-10
- Norway Suspends New Data Center Electricity Reservations Until as Late as August 2027 — Polymarket · 2026-11-10