At what context depth does KV quantization start to hurt? An F16 vs Q8/Q4 parity PoC
Slight_Analysis_5414 · reddit · 2026-09-04
A developer with a CV/YOLO deployment background proposes a new axis for measuring KV-cache quantization damage: not just aggregate KLD, but at what context depth divergence becomes persistent.
The PoC runs identical GGUF weights, tokenizer, and an identical teacher-forced token sequence through F16 vs Q8/Q4 KV caches — isolating precision-induced divergence from ordinary autoregressive branching. It records top1 agreement, top-K overlap, partition KL, truth logprob delta, plus first-mismatch and first-significant-divergence context lengths, resolved by depth (0–128K).
The goal: determine whether distribution-level divergence stays near the repeatability baseline, rises gradually, or spikes at depth — directly relevant to recent Qwen 27B discussions of on-the-fly KV quantization degrading long-context performance.
More from Infra
- Kafka, Kafka Connect and Schema Registry exposed as native MCP tools — jkriket · 2026-09-04
- Conviva Replaced mmap with io_uring in Its Rust Query Engine — and It Got Slower — blaizedsouza · 2026-09-04
- Hugging Face deep dive: browser-compiled GPU kernels, attention in 20 lines of JS at 400 fps — nicodotdev · 2026-09-04
- Hugging Face kernels: Jinja templates let browsers compile the fastest GPU kernels — nicodotdev · 2026-09-04
- New local LLM benchmark tracks prefill speed from RTX 5090 down to Raspberry Pi — maximelabonne · 2026-09-04
- Dev proposes unsecured honeypot compute whose power usage exposes unauthorized training — natesiggard · 2026-09-04