A working vLLM recipe runs nvfp4 KV cache on 2× RTX 5060 Ti
gtrak · reddit · 2026-07-21
A Reddit post shares a working vLLM launch recipe for running nvfp4 KV cache on 2× RTX 5060 Ti with a Qwen3.6-27B-PrismaSCOUT-Blackwell-NVFP4-BF16-vllm model.
What the setup uses
- --kv-cache-dtype nvfp4
- --max-model-len 262144
- --tensor-parallel-size 2
- --kv-offloading-size 20
- --enable-prefix-caching and --enable-chunked-prefill
- --cudagraph-capture-sizes 1 2 4
- --compilation-config {"cudagraphmode":"PIECEWISE"} because FULL reportedly corrupts XQA
Operational notes
- VLLMMEMORYPROFILERESTIMATECUDAGRAPHS=0 reclaims about 0.39 GiB for MTP workspace.
- gpu-memory-utilization=0.90 avoids OOMs; 0.94 reportedly OOMs.
- The post also includes a number of low-level NCCL and multiprocessing flags for stability.
The main value here is a concrete, reproducible serving configuration for squeezing long-context inference and speculative decoding into a consumer GPU setup.
More from Infra
- NeurIPS 2026 workshop will focus on on-device intelligence and local execution — YiMaTweets · 2026-07-21
- How to build a PostgreSQL-backed semantic search pipeline with pgvector and Ollama — KhuyenTran16 · 2026-07-21
- NeurIPS 2026 workshop calls papers on on-device intelligence — YiMaTweets · 2026-07-21
- Milled from Solid Aluminum: AI Rig Multi-GPU Case for Local Compute — dee_hw · 2026-07-21
- FutureCaribbean’s Buildathon offers $50K, H200 compute, and an NYSE pitch — HeyAmit_ · 2026-07-21
- A new series tests which data-science workflows can run on GPUs today — pandeyparul · 2026-07-21