Blind Spot in KV Cache Quantization Eval: 2-bit Matches KIVI
Interesting-Owl6064 · reddit · 2026-07-05
Focusing on KV cache compression for long-context inference on small GPUs, the author found a blind spot in mainstream evaluations: many KV quantization perplexity tests are done during a single forward pass with the cache disabled. Because the model reads full-precision values and the quantizer never actually runs, fp16, 4-bit, and 2-bit all yield the exact same perplexity (3.6416), failing to reflect the value cache's quantization error. The author fixed this by testing via a cache path that does prefill followed by token-by-token decoding.
Methodologically, a Hadamard matrix is used to rotate the value vectors before quantizing them to a 2-bit uniform distribution (rotation disperses outliers, and the self-inverse matrix can be restored for free after the attention sum), while keys still use KIVI int4. Under the corrected metrics, the 2-bit value cache quality matches KIVI 4-bit to three decimal places, reducing VRAM usage by about 20% (roughly 1/4 of fp16). This was validated on Llama 2 7B and TinyLlama, and reproduced on a second machine. Honest limitations: only compared against KIVI (not the latest rotation methods); decoding is 6%–12% slower without a fused kernel.
More from Infra
- LLM Serving Metrics Thread: Why TPOT and Uptime Make or Break User Experience — abhijithneil · 2026-09-11
- PlanetScale launches sharded Postgres: 768 servers acting as one, 1PB scale — dhruv2038 · 2026-09-11
- Can a 7900 XTX 24GB run Qwen locally? Reddit seeks ROCm tok/s benchmarks — thenomadexplorerlife · 2026-09-11
- RTK Terminal Compression Cuts Tokens but Leaves Your AI Coding Bill Unchanged — Bartaseth · 2026-09-11
- SF Compute founder: buying compute is 'an absolutely awful experience' right now — IgorCarron · 2026-09-11
- SmolVM open-sources persistent computer infrastructure for agents that outlive chat sessions — aniketmaurya · 2026-09-11