Counterintuitive: KV cache quantization in llama.cpp uses MORE VRAM than f16

Fancy-Snow7 · reddit · 2026-11-21

A Reddit user running Inkling-2-Mini (Q4KXL quant, 131k context, all experts offloaded to CPU) on the Unsloth build of llama.cpp found that quantizing the KV cache backfires: f16 uses 8.8GB dedicated VRAM, while q80 uses 12.3GB and q40 11.2GB.

Switching to f16 even fixed CUDA out-of-memory errors seen with q80 at higher context settings, and token generation ran up to 50% faster—possibly because freed VRAM allows more experts to be cached. Setup: Windows 11 with an RTX 5070Ti.

Original post →

More from Infra

Infra channel →