Counterintuitive: KV cache quantization in llama.cpp uses MORE VRAM than f16
Fancy-Snow7 · reddit · 2026-11-21
A Reddit user running Inkling-2-Mini (Q4KXL quant, 131k context, all experts offloaded to CPU) on the Unsloth build of llama.cpp found that quantizing the KV cache backfires: f16 uses 8.8GB dedicated VRAM, while q80 uses 12.3GB and q40 11.2GB.
Switching to f16 even fixed CUDA out-of-memory errors seen with q80 at higher context settings, and token generation ran up to 50% faster—possibly because freed VRAM allows more experts to be cached. Setup: Windows 11 with an RTX 5070Ti.
More from Infra
- NVIDIA paper: frequent context compaction cuts LLM serving cost up to 7x vs long context — khademinori · 2026-11-21
- Analyst speculates OpenAI's problems stem from GPU shortage, no SpaceX access — mark_k · 2026-11-21
- Researcher builds local GPU server to run 24/7 AI agent system after Mac mini runs out of memory — nodosenlared · 2026-11-21
- Custom engine runs Inkling 2 Mini at 129.5 tok/s with 344K context on a single RTX 3090, 2.3x faster than llama.cpp — teortaxesTex · 2026-11-21
- Cohere launches confidential computing with encrypted inference; exec says rivals' ZDR labels are marketing — Nils_Reimers · 2026-11-21
- Gary Marcus shares chart of Oracle CDS spiking parabolically to record highs — GaryMarcus · 2026-11-21