KVarN: Variance-Normalized KV-Cache Quantization Mitigates Error Accumulation in Reasoning Tasks
Lorenz K. Muller, Philippe Bich, Chiara Boretti, Hyun-Min Chang, Jiawei Zhuang, Lukas Cavigelli
cs.LG
2026-06-02
A Hadamard rotation plus dual variance normalization fixes token-scale outliers and quantizes the KV cache to 2.3 bits; on Qwen3-4B AIME24 KVarN hits 60.0%, near FP16's 61.1% and well above KIVI.
Test-time scaling produces better LLM reasoning but creates a memory bottleneck: the KV cache grows with long decoding. KV-cache quantization to 2 bits can help, but existing methods are evaluated under "prefill-like" settings where a fixed long context is quantized in parallel. Under autoregressive decoding, generated tokens are compressed on the fly, and quantization errors accumulate across timesteps as early errors propagate through later transformer-block outputs. Existing methods do not preserve per-token scales, so the errors compound.
KVarN decomposes the quantization error into a magnitude term and a directional term, and shows that outlier errors are mostly magnitude (wrong token scaling), not directional. Its pipeline at 2.3 bits per element is: a Hadamard rotation in the channel dimension, then dual-scaling variance normalization (Sinkhorn-Knopp-style, over both token and channel dimensions), then round-to-nearest (RTN) 2-bit quantization.
Variance normalization fixes the token-scale outliers with a high-precision per-group scale, which stops the wrong multiplier from compounding across timesteps. Storage is 2-bit values plus an FP8 scale and an FP16 zero point per 128-element group; tile size is 128x128.
| Method | Qwen3-4B AIME24 | Phi-4-14B AIME24 | Bits/elem |
| FP16 | 61.1 | 62.2 | 16.0 |
| KIVI | 55.5 | 57.8 | 2.3 |
| KVQuant-1% | 40.0 | 55.6 | 2.4 |
| KVarN | 60.0 | 61.7 | 2.3 |
On MATH500, KVarN reaches 79.2 on Qwen3-4B (FP16 82.6) and 84.8 on Phi-4-14B (FP16 84.9, essentially lossless). On 600-line retrieval, KVarN scores 85% versus KIVI's 74% and FP16's 90%. Memory drops from 16 to 2.3 bits per element, an 85.6% reduction, with overhead of about 1.9ms per 128 tokens (0.18%).
Long chain-of-thought reasoning models burn memory on the KV cache; KVarN compresses it about 7x at near-lossless accuracy where prior 2-bit methods collapse, especially on math (AIME24) and long retrieval. It integrates with vLLM. The caveat is timing: the gains are partly in principle until inference engines support 2-bit KV end to end.
KVarN only helps architectures with a KV cache; it does not apply to state-space models (SSMs). Its applicability to MLA (DeepSeek-style multi-head latent attention) is unclear. No current serving framework supports 2-bit KV caches end to end, so the full benefit waits on inference-engine support.