Q8 KV cache quantization hurts long-context — because it quantizes on every write
maddie-lovelace · reddit · 2026-08-28
Experiments on Qwen3.8-27B suggest Q8 KV cache quantization is not free: backends like llama.cpp quantize KV on write, so every prefill step reads quantized keys, and the sub-1% rounding error compounds across every layer. Needle retrieval that passes at bf16 fails with q8-on-write at 125k context.
Crucially, the problem is when quantization happens, not quantization itself: taking a bf16-built cache and quantizing it wholesale to q8 introduces only the expected 1% error, and retrieval works fine again. Caveats: single model family, few trials, and some runs on a custom MLX stack — but the mechanism likely generalizes.
More from Infra
- US Commerce Dept to Block Chinese Access to Overseas NVIDIA Chips — McDonaghMatthew · 2026-08-28
- US data centers face 17-29 GW power load gap by 2026 — BenBajarin · 2026-08-28
- AI capex chases adoption, not leads it — YvesMulkers · 2026-08-28
- a16z on Why Infrastructure Is AI's New Bottleneck — a16z · 2026-08-28
- AI Chipmaker Sunrise Doubles Valuation After Raising 2 Billion Yuan — pstAsiatech · 2026-08-28
- a16z raises $1.1B Machine Age Fund to build AI infrastructure — a16z · 2026-08-28