SGLang ships NVFP4 KV cache: up to 78% faster long-context decode on Blackwell
BanghuaZ · x · 2026-09-22
- LMSYS, together with Qwen (Alibaba) and NVIDIA, brought NVFP4 4-bit KV cache to SGLang on Blackwell GPUs, targeting long-context and agentic inference.
- NVFP4 stores KV at 56% of FP8's footprint per token, fitting 1.78x more context in GPU memory.
- Decode throughput improves +37% / +58% / +78% at 32K / 160K / 1M context lengths.
- Near-lossless accuracy: matches FP8 on GPQA-Diamond and AIME 2025 with Qwen3.5-397B-A17B; higher cache hit rates keep AgentX throughput scaling where FP8 drops off.
- Built from NVFP4 two-level scaling, in-kernel dequantization on decode, and paged KV cache; enabled in SGLang with a single flag --kv-cache-dtype nvfp4.
More from Infra
- Omarchy 4.6 lets you share local AI models across your tailnet, says DHH — alexcovo_eth · 2026-11-12
- Eric Schmidt's foundation funds French NGO fighting against data centers — StephanSturges · 2026-11-12
- Epoch: Open-Weight GLM 5.3 Hacks Six Firefox Versions for ~$1,000 Each; Claude Fable 5.2 Sets ECI Record — Epoch AI · 2026-11-12
- Polymarket prices 28% odds of a state data center moratorium; Torus raises $500M — Polymarket · 2026-11-12
- AWS-wrapper cloud startup shuts down just one month after seed round — thedealdirector · 2026-11-12
- Utah startup Torus raises $500M for shipping container-sized AI data centers deployable in 4 months — Polymarket · 2026-11-12