NVIDIA paper shows cross-model KV cache transfer skipping prefill, 2.7–25x faster than recomputing
solyarisoftware · x · 2026-10-19
An NVIDIA research paper introduces cross-model KV cache transfer: models in the same family (e.g. Qwen3 14B→32B) reuse another model's KV cache and skip prefill entirely, with conversion running 2.7–25x faster than reprocessing the context.
Context: LLM APIs are stateless, so every turn resends the full conversation billed as input. Prompt caching gives 90% discounts on cache hits but only works for the model that produced the cache — a major waste for routing and cascading across model sizes.
Key findings:
- Cross-model KV has strong linear structure: one source layer explains 56% of variance in the target's keys and 32% in values on Qwen3 14B→32B, rising to 79%/65% with multiple source layers.
- A closed-form ridge mapper operates per head in three steps: select top-k predictive source layers per target layer, strip RoPE from keys before mapping, then reconstruct the cache.
Implication: cost-quality cascades, mid-conversation model switching, and routing no longer need to repay prefill on every swap.
More from Infra
- DFlash2 authors ship Qwen3.8-Flash speculative decoding model, 20% faster than MTP at concurrency 1 — rerri · 2026-10-19
- DOE lends $4.4B to restart three nuclear plants to feed AI data centers — emmanuelvivier · 2026-10-19
- China proposes export controls on 23 AI technologies, banning 2T-parameter models — emmanuelvivier · 2026-10-19
- DeepSeek-V4.1-Flash quantized to 2-bit runs on two RTX PRO 6000s, KL eval published — TheZachMueller · 2026-10-19
- Alibaba Cloud shows Omarchy running on Wuying AI cloud PCs for enterprise agents — AIFlow_ML · 2026-10-19
- pi-durable v1.3.0 adds SQLite table prefixes, fixes 140 MB memory spike on DO — solyarisoftware · 2026-10-19