Dev hacks llama.cpp for NVFP4 KV cache, runs Qwen3 27B at 262k context across two GPUs
comperr · reddit · 2026-09-19
A Reddit user got NVFP4 KV cache working in a patched llama.cpp (heterogeneous port of an ollama branch), running a self-made Q4KM quant of Qwen3 27B at 262k context split across an RTX 5090 and RTX 3090 Ti. Key technical points:
- Compute-aware layer dispatch: FA kernels split by CUDAARCH, not just FP16 capability — separate SM86 and SM120 VEC paths, output layer locked to the 5090
- Real bug fixes: q80 TILE boundary nwarps overflow at 262k context (now passes on both cards), plus misaligned vector loads for MXFP4 17B and NVFP4 8B blocks on SM120
- Core fix: Blackwell can't vector-load 17-byte blocks, so use byte-wise memcpy (memcpy(&k, K+off17, 17))
- Weights are a custom Q4KM repacked from the fp16 source
Directly useful for anyone doing long-context local deployment or quantization work on consumer multi-GPU setups.
More from Infra
- Malaysia poised to become a compute superpower with both US and Chinese chips by next year — basedjensen · 2026-11-04
- ByteDance details USDB storage system: 85.8% lower write latency for recommender sequences — _reachsumit · 2026-11-04
- Oracle cancels $7B chip leasing deal with Tencent, FT reports — SumitGup · 2026-11-04
- Inference engine shrunk to 1023 bytes, still runs SmolLM2-135M — cneuralnetwork · 2026-11-04
- Musk says space data centers are 'looking pretty appealing right now' — elonmusk · 2026-11-04
- Data centers are turning to old jet engines as the wait for power grows — IndependentCoast3974 · 2026-11-04