SGLang ships NVFP4 KV cache: up to 78% faster long-context decode on Blackwell

BanghuaZ · x · 2026-09-22

• LMSYS, together with Qwen (Alibaba) and NVIDIA, brought NVFP4 4-bit KV cache to SGLang on Blackwell GPUs, targeting long-context and agentic inference.
• NVFP4 stores KV at 56% of FP8's footprint per token, fitting 1.78x more context in GPU memory.
• Decode throughput improves +37% / +58% / +78% at 32K / 160K / 1M context lengths.
• Near-lossless accuracy: matches FP8 on GPQA-Diamond and AIME 2025 with Qwen3.5-397B-A17B; higher cache hit rates keep AgentX throughput scaling where FP8 drops off.
• Built from NVFP4 two-level scaling, in-kernel dequantization on decode, and paged KV cache; enabled in SGLang with a single flag --kv-cache-dtype nvfp4.

Original post →

More from Infra

• SemiAnalysis: mapping MoE models onto inference hardware — zephyr_z9 · 2026-09-22
• Huawei's HiZQ is real HBM; DeepSeek's designs all target bandwidth, not storage — zephyr_z9 · 2026-09-22
• Halo RL training platform launches with SGLang as primary rollout engine — ying11231 · 2026-09-22
• SiliconBench: speed, memory and fidelity of nine LLM engines on unified-memory desktops — PennState · 2026-09-22
• Engram retrieval won't replace FFNs, but cutting 40-50% of HBM needs is the real win — bookwormengr · 2026-09-22
• Huawei's Sept 19 event: long-running agents bottleneck on data movement, not FLOPs — krishnan · 2026-09-22

Infra channel →