Blind Spot in KV Cache Quantization Eval: 2-bit Matches KIVI
Interesting-Owl6064 · reddit · 2026-07-05
Focusing on KV cache compression for long-context inference on small GPUs, the author found a blind spot in mainstream evaluations: many KV quantization perplexity tests are done during a single forward pass with the cache disabled. Because the model reads full-precision values and the quantizer never actually runs, fp16, 4-bit, and 2-bit all yield the exact same perplexity (3.6416), failing to reflect the value cache's quantization error. The author fixed this by testing via a cache path that does prefill followed by token-by-token decoding.
Methodologically, a Hadamard matrix is used to rotate the value vectors before quantizing them to a 2-bit uniform distribution (rotation disperses outliers, and the self-inverse matrix can be restored for free after the attention sum), while keys still use KIVI int4. Under the corrected metrics, the 2-bit value cache quality matches KIVI 4-bit to three decimal places, reducing VRAM usage by about 20% (roughly 1/4 of fp16). This was validated on Llama 2 7B and TinyLlama, and reproduced on a second machine. Honest limitations: only compared against KIVI (not the latest rotation methods); decoding is 6%–12% slower without a fused kernel.
More from Infra
- WSJ: Nvidia is in talks to backstop about $250 billion of OpenAI's data center plan — KateClarkTweets · 2026-07-27
- YC talk on BCI x AI says infrastructure is what really determines speed — garrytan · 2026-07-27
- A 13B model ran on a no-GPU PC by paging weights from SSD via llama.cpp — ID_R_McGregor · 2026-07-27
- llama.cpp warns that GGUFs made before a recent change must be regenerated — EconomySerious · 2026-07-27
- Surprising Ubuntu Setup: NVIDIA 5090 PC Becomes the Easiest AI Rig — _xjdr · 2026-07-27
- TSMC reportedly plans 5%–10% price hikes in 2027 to cover rising costs — Beth_Kindig · 2026-07-27