How SGLang's RadixAttention reuses prefix KV cache to boost agent inference up to 6.4x

大模型之路 · wechat · 2026-09-15

This article breaks down a hidden cost in LLM inference: agent loops, RAG pipelines, and multi-turn chats keep re-sending the same system prompts, tool definitions, and retrieved documents, forcing GPUs to recompute identical KV caches at scale. SGLang's RadixAttention stores all KV caches in a radix tree keyed by token sequences, automatically matching and reusing the longest shared prefix — token-granular matching that handles irregular overlaps better than vLLM's chunked PagedAttention hashing, with up to 6.4x throughput gains reported on agent and JSON decoding workloads. It adds tiered caching (offloading cold prefixes to CPU RAM/NVMe), prefix-aware batching, constrained decoding, and zero-overhead CPU scheduling; industry estimates put it on 400K+ GPUs at xAI, NVIDIA, LinkedIn and others. The key decision rule: if your traffic has structural prefix redundancy, SGLang wins big; if requests are all unique single-shot generations, it roughly ties vLLM. Pitfalls include one model per server process, radixcache memory limits, and timestamps/random IDs breaking prefix alignment. Setup is a pip install and one launch command exposing an OpenAI-compatible endpoint, with FP8/INT4/AWQ/GPTQ quantization support.

Original post →

How SGLang's RadixAttention reuses prefix KV cache to boost agent inference up to 6.4x

More from coding & agent