Dev hacks llama.cpp for NVFP4 KV cache, runs Qwen3 27B at 262k context across two GPUs
comperr · reddit · 2026-09-19
A Reddit user got NVFP4 KV cache working in a patched llama.cpp (heterogeneous port of an ollama branch), running a self-made Q4KM quant of Qwen3 27B at 262k context split across an RTX 5090 and RTX 3090 Ti. Key technical points:
- Compute-aware layer dispatch: FA kernels split by CUDAARCH, not just FP16 capability — separate SM86 and SM120 VEC paths, output layer locked to the 5090
- Real bug fixes: q80 TILE boundary nwarps overflow at 262k context (now passes on both cards), plus misaligned vector loads for MXFP4 17B and NVFP4 8B blocks on SM120
- Core fix: Blackwell can't vector-load 17-byte blocks, so use byte-wise memcpy (memcpy(&k, K+off17, 17))
- Weights are a custom Q4KM repacked from the fp16 source
Directly useful for anyone doing long-context local deployment or quantization work on consumer multi-GPU setups.
More from Infra
- 16GB VRAM Users Be Like — tassa-yoniso-manasi · 2026-09-19
- antirez: API pricing is Monopoly money — the only real metric is joules, and we can't see them — mitsuhiko · 2026-09-19
- Domestic micro-datacentres strapped to water tanks cut UK heating bills by £10-15/month — nordicinst · 2026-09-19
- DiffusionGemma-based DJev runs near-real-time vision detection on a phone — PMinervini · 2026-09-19
- 35B ternary LLM runs on iPhone in ~4GB RAM: Millie beats Bonsai 27B, 60% faster — MannyKayy · 2026-09-19
- PyTorch Conference to showcase CRCR, letting downstream repos plug into upstream CI — PyTorch · 2026-09-19