LowRAM: Low-Rank Keys and Amortised Values for Composable KV Cache Compression at Test-Time
Teresa Delgado, Hugo Hazard, Martin Benfeghoul, Jun Wang, Zafeirios Fountas
cs.LG, cs.CL
2026-11-05
A per-head MLP fitted at test time predicts KV values from keys, storing residuals only for the worst tokens: 4× near-lossless, 26.9× when stacked with quantisation and eviction.
The KV cache is what keeps autoregressive decoding cheap: key and value projections are computed once per token and reused at every step. The price is memory that grows linearly with context length and batch size, which caps concurrency and throughput at long context. Architectural alternatives (linear attention, latent attention) require retraining, while most open-weight models still ship grouped-query attention with a full cache, so post-training compression remains the practical route. Existing methods come in three families: eviction discards unimportant tokens, quantisation lowers precision, low-rank factorisation compresses the matrices. Each family works a different redundancy axis, and all of them treat keys and values as the same kind of object. This paper finds an axis nobody had systematised: within a single attention head, values are largely predictable from their own keys, because both are projections of the same hidden state.
The design rests on a source-target analysis: at a fixed 3× budget, current-layer keys or values are reconstructed from various sources (previous-layer keys, previous-layer values, both concatenated, current-layer keys), and the winner is chosen by recovered LongBench score. Current-layer key to current-layer value wins. A side finding shapes the whole method: reconstruction error correlates only weakly with downstream performance, so every later decision is made on benchmark score.
The value side, called value cache amortisation:
Keys are not predicted. They are harder to reconstruct, and a key error perturbs the attention logits and can pull attention onto the wrong token, whereas a value error lands only after the attention weights are fixed. Key energy also concentrates in a few directions, so keys go through xKV, a cross-layer low-rank SVD that shares one basis across adjacent layers, with RoPE inverted before compression. Decode overhead is handled by selective reconstruction borrowed from ShadowKV: the prefix is chunked with mean-key landmarks, each step reconstructs only the top-k chunks selected by the current query, and a randomised Cholesky-QR replaces exact SVD for a 4× speedup at no quality cost.
End-to-end compression, recovered performance as a percentage of the uncompressed baseline:
| Setting | Mistral RULER | Mistral LongBench | Llama RULER | Llama LongBench |
| 3× | 98.8 | 99.4 | 97.9 | 98.5 |
| 4× | 96.0 | 97.9 | 93.9 | 99.6 |
Against the standalone families (Mistral, LongBench):
| Method | Compression | Recovery |
| Quantisation (TurboQuant) | 5.1× | 98.9 |
| LowRAM | 4× | 97.9 |
| LowRAM-Q | 6.4× | 99.4 |
| LowRAM-EQ | 10.7× | 96.9 |
Quantisation stacks well: cheaper residual storage buys more residuals, lifting Llama RULER at 4× from 93.9% to 99.4%. Eviction stacks worse, since it shortens the sequence without shrinking the predictor's fixed cost. At the extreme (LongBench subset above 10K tokens, maximum compression retaining 90% of baseline): eviction 10×, quantisation 7.5×, LowRAM alone 9.0×, and the full LowRAM-EQ stack 26.9× at 94.5% recovery, ahead of any single family. On efficiency, the meta-initialisation cuts TTFT overhead from 26.3× baseline (150 updates) to 1.3× (2 updates); at 4× compression and 262K tokens, selective reconstruction reaches roughly 5× the baseline's peak decode throughput, at a batch size of 28 where the baseline has already gone out of memory. Two choices carry the method in ablations: removing the residual store drops LongBench recovery from 99.0% to 64.9%, a purely linear predictor falls to 57.5%, and sharing one predictor across a layer's heads costs 19.5 points. Residual selection matters as much: on a needle-in-haystack task, random selection scores under 40% against 98% for error-ranked selection.
This is a new redundancy axis orthogonal to the three established families, and it is post-training: no retraining, no change to attention, directly applicable to existing GQA checkpoints. Three practical payoffs: saved memory converts into batch size and throughput; it stacks with quantisation and eviction pipelines already in production; and in shared-prefix serving (document QA, many queries over one context) the expensive full fit is paid once. On its own, 4× near-lossless is comparable to good quantisation. The increment is composability: 26.9× is not a number any single family reaches.
The authors' own list: long generations are unevaluated, since that would need periodic re-compression of the generated suffix rather than a single prefill-time fit, and it is unclear how much of the learned key-to-value map transfers across contexts. The structural cost is stated plainly: the predictor's parameters are a fixed overhead that dominates below roughly 2-3K tokens, which is why RULER 4K degrades more than LongBench and why eviction composes poorly at high ratios (Llama LowRAM-E at 8× keeps only 79.9% on RULER). The cheap configuration has a price: meta-initialisation costs 1.4-4.4 points on RULER versus 300-update fitting, and with selective reconstruction RULER 64K lands at 92.4%.
Points to watch when reading: 26.9× depends on quantisation and eviction stacked on top, measured on the >10K subset with a 90% retention threshold, not LowRAM alone; the 1.3× TTFT figure covers value amortisation only and the full pipeline with key factorisation is higher; Qwen3.5-4B is a hybrid architecture whose ratios apply to its full-attention layers only; there is no end-to-end comparison against AQUA-KV, the closest predictive method; and evaluation stops at 8B models.