Cross-Model KV Cache Transfer in LLM Families: A Closed-Form Linear Mapping for Prefill Reuse
Taekyung Heo, Rasoul Shafipour, Ritchie Zhao, Maximilian Golub, Mohammad Mahdi Kamani, Ritika Borkar, Makesh Tarun Chandran, Pantea Zardoshti, Bita Darvish Rouhani
cs.LG
2026-08-05
NVIDIA maps one model's KV cache onto another in the same family to skip re-prefill; the gradient-free ridge fit holds 73-98% accuracy on 4 of 6 pairs at up to 25x lower latency.
Production LLM serving routinely swaps between different-sized models in a family: a cheap small model handles the easy traffic and escalates the hard cases to a larger one (cost-quality cascading), conversations switch models mid-stream, or a router picks the right tier per query. Every swap forces the receiver to replay the full context through all its layers and recompute its KV cache, a stage called prefill. With long contexts, prefill is the expensive part.
Prior KV-reuse work needed either a trained fusion network per model pair (C2C), adapters projecting both sides into a shared latent space (LatentAlign), or identical architectures (DroidSpeak). This paper skips gradient training entirely and moves the source model's KV cache onto the target with a closed-form linear map.
The mapper is a per-head ridge regression, built in three steps:
The constraint is matched-KV: source and target must share KV head count and per-head dimension (aligned GQA structure). Layer count and parameter count can differ, so this works within a family but not across families.
Why linear is enough: KV caches across model scales in a family are strongly linearly related. On Qwen3 14B→32B, a single source layer explains 56% of variance in the target's keys and 32% in values; multiple source layers raise that to 79% (K) and 65% (V), and the best single cell reaches R²=0.81.
Six pairs across three families (Qwen3, Llama 3.1, Ministral 3), accuracy retention on five benchmarks (the target's standalone-prefill accuracy is 100%):
| Pair | Avg retention | HellaSwag | MMLU | GSM8K |
| Qwen3 14B→32B | 97.5% | 97.6% | 95.0% | 95.6% |
| Qwen3 8B→32B | 87.5% | 95.2% | 88.5% | 68.8% |
| Llama 3.1 8B→70B | 72.8% | 94.4% | 73.3% | 18.2% |
| Ministral 3B→8B | 76.2% | 93.3% | 69.4% | 36.6% |
| Ministral 3B→14B | 44.2% | 68.0% | 32.0% | 3.2% |
| Ministral 8B→14B | 41.6% | 58.7% | 32.7% | 1.6% |
Four pairs hold 73–98% average accuracy; two (Ministral 3B→14B and 8B→14B) collapse to 42–44%. GSM8K is the tell: even on the successful pairs, Llama 3.1 8B→70B retains only 18.2%, under a fifth of the target's standalone math ability. Multiple-choice and commonsense tasks tolerate KV transfer because they are insensitive to activation precision; precise reasoning exposes it.
Speed. On Qwen3 14B↔32B, small-to-large prefill latency drops 4–25×, from 6.98s to 0.28s at 32K tokens; large-to-small drops 3–7×. The mapper is faster in all 70 tested cells. The mapper itself is not small: 1.0–3.4B params, 4–12GB of storage.
Nonlinear backstop. On the two failing pairs, swapping ridge for a two-layer MLP (1,024 units each) lifts HellaSwag retention from 68.0%/58.7% to 92.3%/95.5%, a 24–37-point gain. Where ridge already works, the MLP is marginally worse. The authors attribute the gain to the MLP pushing residual error out of attention-sensitive subspaces.
Teams running model cascades (a small model fields traffic, escalates hard cases to a larger model, or routes per query) can use this to skip the full prefill replay on every swap. For long contexts (tens of thousands of tokens), that is seconds of latency and the matching compute saved.
The "matters" needs an asterisk. Three limits bite: it works only within a family (matched-KV required), it is validated only on dense full-attention, and math-reasoning tasks degrade even on the successful pairs. The genuinely safe fit is same-family scale swaps on precision-insensitive workloads (classification, summarization, commonsense QA, multi-turn chat). The authors checked multi-turn on CoQA: Qwen3 14B↔32B over 10 turns accrues a 1.7-point gap small-to-large and a linear 0.33-point drift per turn large-to-small, with no cascade failure inside ten turns, though large-to-small drift keeps accumulating over very long sessions.
The authors concede: calibration uses only FineWeb-Edu; switching to CodeAlpaca costs 5.24 points on HellaSwag (Wikipedia stays within noise); only matched-KV pairs are tested, mismatched ones are untouched; only dense full-attention is covered, so hybrid architectures with sliding-window or sparse attention are out of scope.
Two things stand out on a close read. First, the natural a priori metric (calibration R²) does not predict downstream quality: Llama 3.1 8B→70B fits well yet degrades downstream. Attention-output cosine turns out to predict retention (Pearson r=+0.57 versus R² at −0.20), meaning where the error lands matters more than how large it is. That is a sharp observation, but it also means you cannot tell at calibration time whether a pair is worth doing; you find out only after running the downstream benchmarks. Second, the "73–98% accuracy" headline is an optimistic summary once GSM8K is in view, since math collapses across the board; in deployment the numbers have to be read per task type.