Q8 KV cache quantization hurts long-context — because it quantizes on every write
maddie-lovelace · reddit · 2026-08-28
Experiments on Qwen3.8-27B suggest Q8 KV cache quantization is not free: backends like llama.cpp quantize KV on write, so every prefill step reads quantized keys, and the sub-1% rounding error compounds across every layer. Needle retrieval that passes at bf16 fails with q8-on-write at 125k context.
Crucially, the problem is when quantization happens, not quantization itself: taking a bf16-built cache and quantizing it wholesale to q8 introduces only the expected 1% error, and retrieval works fine again. Caveats: single model family, few trials, and some runs on a custom MLX stack — but the mechanism likely generalizes.
More from Infra
- Copper to optical: optical backplanes seen as necessary as racks move to optics — BenBajarin · 2026-10-16
- Polymarket prices 28% odds a US state enacts data center moratorium by year-end — Polymarket · 2026-10-16
- Pennsylvania residents reject $10,000-per-household offer for massive Amazon data center — Polymarket · 2026-10-16
- PyTorch now runs natively on AWS Trainium with zero code changes — PyTorch · 2026-10-16
- Chinese labs run on ~1/10 US compute; ByteDance pays 2-4x to hoover up SEA supply — robleclerc · 2026-10-16
- Zymtrace raises $12.2M and hires ML engineers to build agents that fix GPU bottlenecks — cfregly · 2026-10-16