SGLang Reserves 18.5GB for GDN State, vLLM Doesn't: 5x KV Cache Gap on Qwen3.8-27B
SomeRandomGuuuuuuy · reddit · 2026-08-18
A Reddit user benchmarked vLLM v0.27.1 vs SGLang 0.5.17 on Qwen/Qwen3.8-27B (BF16, TP=1) on an RTX PRO 6000 Blackwell (97GB), with matched settings (256K context, mem fraction 0.8, 128 concurrent requests), and found a stark difference:
- SGLang reserves 18.49 GB for recurrent GDN state across 128 request slots, leaving only 74,460 KV tokens
- vLLM exposes 356,764 KV tokens — about 4.8x more
The model is hybrid attention: of 64 layers, 16 are full-attention and 48 are linear-attention/GDN layers (4 KV heads, headdim 256). The author aligned flags as closely as the two APIs allow (chunked prefill, cudagraph, prefix cache off), but still cannot explain the gap, and recalls seeing something similar six months ago. He asks whether this is expected behavior or a misconfiguration.
More from Infra
- First AA-AgentPerf results: NVIDIA Vera Rubin NVL72 delivers up to 5.7x throughput per MW vs GB300 — ArtificialAnlys · 2026-10-16
- Vera Rubin NVL72 first benchmarks: up to 5.7x throughput per MW vs GB300 — ArtificialAnlys · 2026-10-16
- NVIDIA Vera Rubin NVL72 hits up to 5.7x GB300 throughput per MW on agent benchmark — ArtificialAnlys · 2026-10-16
- 14 open-source LLM observability tools to know in 2026, from Langfuse to Laminar — TheTuringPost · 2026-10-16
- Local AI is a routing problem, not a privacy feature: where each task should run — bigdata · 2026-10-16
- Moving agent work from GPUs to CPUs for durable, predictable actions — cramforce · 2026-10-16