Two vLLM recipes for Blackwell: NVFP4 KV cache buys 262K context and more streams

SeanHighness · reddit · 2026-08-27

The author released two vLLM recipes for Blackwell GPUs, built around the idea that the KV cache — not just model weights — should use Blackwell's native FP4 capability.

Why NVFP4 KV matters: KV cache is one of the biggest VRAM consumers at long context/high concurrency. Compressing it from BF16/FP8 to FP4 frees VRAM for longer contexts, more concurrent streams, and larger KV pools. On a 5090, that's the difference between barely fitting 262K context and serving several useful streams at once.

Two builds:

MTP-3 (capacity-first): 400K usable KV pool, up to 8 streams, 262K max context — for multi-agent/multi-user/long-context workloads;
DFlash2 (speed-first): 325K KV pool, up to 4 streams, substantially faster speculative decode — strong for single-stream, low-concurrency use.

Extras: the vision tower runs in a CPU sidecar, preserving multimodal input while keeping GPU VRAM for serving. The builds combine NVFP4 weights + NVFP4 KV + MTP/DFlash2 speculative decoding + Blackwell kernels — combinations not yet in upstream vLLM — packaged as reproducible builds from community PRs plus the author's fixes.

Who it's for: FP4-native Blackwell hardware (RTX 5090, 2×16GB Blackwell, RTX Pro) running sub-agents, multi-user inference, or parallel coding agents. Production users wanting official support should stick with vanilla vLLM.

Original post →

Two vLLM recipes for Blackwell: NVFP4 KV cache buys 262K context and more streams

More from coding & agent