SGLang Reserves 18.5GB for GDN State, vLLM Doesn't: 5x KV Cache Gap on Qwen3.8-27B
SomeRandomGuuuuuuy · reddit · 2026-08-18
A Reddit user benchmarked vLLM v0.27.1 vs SGLang 0.5.17 on Qwen/Qwen3.8-27B (BF16, TP=1) on an RTX PRO 6000 Blackwell (97GB), with matched settings (256K context, mem fraction 0.8, 128 concurrent requests), and found a stark difference:
- SGLang reserves 18.49 GB for recurrent GDN state across 128 request slots, leaving only 74,460 KV tokens
- vLLM exposes 356,764 KV tokens — about 4.8x more
The model is hybrid attention: of 64 layers, 16 are full-attention and 48 are linear-attention/GDN layers (4 KV heads, headdim 256). The author aligned flags as closely as the two APIs allow (chunked prefill, cudagraph, prefix cache off), but still cannot explain the gap, and recalls seeing something similar six months ago. He asks whether this is expected behavior or a misconfiguration.
More from Infra
- Grid Bottlenecks Stall AI: Interconnection Queues Surge to 45 Months — PeterDiamandis · 2026-08-18
- Cooling solutions for multi-3090 setup for local inference in 2026 — Sevealin_ · 2026-08-18
- Nvidia secures 35%-40% of global HBM supply for next year — JOBhakdi · 2026-08-18
- pagedMark: Invisible SynthID watermark removal optimized for Apple Silicon — d0ofz · 2026-08-18
- Ex-SpaceX engineers build AI robotic factory for steel parts — Ars Technica AI · 2026-08-18
- Benchmarking Qwen3.8-27B on 4x RTX 3090: Topology Matters — Mr_Moonsilver · 2026-08-18