Grouped Value Attention: Efficient KV Caching via On-Demand Key Reconstruction
Vishesh Tripathi, Abhay Kumar, Ramsha Khan
cs.AR, cs.LG
2026-09-09
GVA caches grouped values and reconstructs keys with a query-absorbed map plus short shared RoPE. At 350M it scores 44.35 vs GQA 44.36, with 45–47% fewer cache scalars.
Autoregressive decoding rereads the full KV cache at every step. Once context grows, that cache dominates both capacity and memory bandwidth. Grouped-query attention (GQA) lets several query heads share one key-value group, shrinking the cache from one pair per head to one pair per group. Each position still writes two streams: a key and a value. Multi-head latent attention (MLA) compresses those streams into a joint low-rank latent. The cache gets smaller; the decode path gets more involved.
Grouped Value Attention (GVA) makes a simpler bet: the value already carries the content that attention will mix into the output, so the content key can be a learned linear function of that value. A second content-key cache is then unnecessary.
GVA keeps GQA grouping: H query heads share G value heads. Each query head has its own reconstruction map Mh, and the content key is Kh = Vg(h) Mh. At decode time Mh is frozen, so the content score q·k equals an inner product between a once-transformed query and the cached value. Content keys never have to be materialized. That identity is exact; the paper calls it absorption.
Vanilla RoPE breaks absorption. The rotation depends on the cached position j and cannot fold into one query reused across the whole cache. GVA copies MLA's split: an unrotated content slice reconstructed from V, plus a short shared positional key of width dr that is cached already rotated. Position survives without restoring a full key cache. Because the positional stream is shared across heads, it costs dr scalars per token, not G·dr.
Setting K=V was the first cut. It halves the GQA cache and never recovers GQA's training loss. One vector cannot both score and be retrieved. Default initialization of M created a second failure mode: keys started far smaller than queries, softmax went nearly uniform, and early training was spent repairing scale. Initializing M so that keys match the query RMS, σM = σQ/(σV√din), puts GVA on a GQA-like loss curve.
Every comparison is a 350M-class decoder trained from scratch on 30B FineWeb-Edu tokens with the same data order, optimizer, and context length. Downstream numbers are zero-shot accuracy on HellaSwag, WinoGrande, OpenBookQA, ARC-Easy, and ARC-Challenge, each configuration averaged over three seeds.
| Method | Five-task average |
| GQA | 44.36 |
| MLA | 43.88 |
| GVA, default init | 43.91 |
| GVA, scale-matched + query RMSNorm | 44.41 |
| GVA + decoupled RoPE, dr=16 | 44.35 |
| GVA + decoupled RoPE, dr=24 | 44.29 |
The intended serving design is GVA with decoupled RoPE. At dr=16 the average is 44.35, 0.01 points below GQA, with about 47% fewer persistent cache scalars than matched GQA (about 45% at dr=24). Those are representation counts, not measured peak serving memory. The scale-matched row edges GVA to 44.41, slightly above GQA, but that path still applies ordinary RoPE to the reconstructed key and cannot absorb M.
Shared KV (K=V) stays above GQA in loss for the whole plotted run. The paper treats it as a failed first cut, not a deployable baseline.
This is a GQA-shaped serving option that stores the value used in the weighted sum, with no extra latent projection on the cache path. Token-by-token decode that is bandwidth-bound should in principle read less. The paper is explicit that prefill is still matrix-heavy, so GVA may look like GQA there, and that custom decode kernels are still being timed. No latency or tokens-per-second number appears.
If a stack already runs GQA, this is a representation swap, not an MLA-style up-projection rewrite. Quality at 350M sits in the same band. Whether it is worth shipping waits on the kernel numbers.
The authors list the gaps themselves. The 45–47% cut describes the intended decode state (grouped V plus shared krope); fused throughput, peak serving memory, and batch capacity are unreported. All matched runs sit at one scale (350M), one mix (30B FineWeb-Edu), and three seeds. There is no systematic sweep of RoPE width, how content width is carved, or longer contexts. The dr=24 row already trails GQA on the average.
Content keys remain linear maps of grouped values, not independent projections. Head-specific scoring comes back; the representation ceiling is still that shared value. Larger models, long context, and real serving curves are not in this paper.