Blind Spot in KV Cache Quantization Eval: 2-bit Matches KIVI
Interesting-Owl6064 · reddit · 2026-07-05
Focusing on KV cache compression for long-context inference on small GPUs, the author found a blind spot in mainstream evaluations: many KV quantization perplexity tests are done during a single forward pass with the cache disabled. Because the model reads full-precision values and the quantizer never actually runs, fp16, 4-bit, and 2-bit all yield the exact same perplexity (3.6416), failing to reflect the value cache's quantization error. The author fixed this by testing via a cache path that does prefill followed by token-by-token decoding.
Methodologically, a Hadamard matrix is used to rotate the value vectors before quantizing them to a 2-bit uniform distribution (rotation disperses outliers, and the self-inverse matrix can be restored for free after the attention sum), while keys still use KIVI int4. Under the corrected metrics, the 2-bit value cache quality matches KIVI 4-bit to three decimal places, reducing VRAM usage by about 20% (roughly 1/4 of fp16). This was validated on Llama 2 7B and TinyLlama, and reproduced on a second machine. Honest limitations: only compared against KIVI (not the latest rotation methods); decoding is 6%–12% slower without a fused kernel.
More from Infra
- Resin lets an agent build standalone LLM inference engines; vLLM and SGLang optimize DeepSeek kernels differently — AccBalanced · 2026-10-27
- Local benchmark: Flint matches Qwen3.6 quality at 50/52 while running 3.6x faster on M4 Pro — RichardSocher · 2026-10-27
- Anthropic S-1: CSP marketplace revenue jumped from 11% to 48% in three years — AccBalanced · 2026-10-27
- Alchemy adds effect/Workflow support across Durable Objects, Workflows and Lambda Durable Functions — samgoodwin89 · 2026-10-27
- Alchemy's Lambda Web Functions run plain Node web servers on Lambda, no OpenNext needed — samgoodwin89 · 2026-10-27
- Alchemy Cuts Cloudflare Bindings CPU Time by Up to 166× Via Profiling — samgoodwin89 · 2026-10-27