NVIDIA's Cross-Model KV Cache Transfer Speeds Up Inference by 25x
theomitsa · x · 2026-08-07
NVIDIA researchers introduced a cross-model KV cache transfer technique. When swapping between different-sized models in an LLM family (e.g., Qwen3 14B to 32B), this method allows the target model to reuse the source's KV cache, entirely skipping the prefill phase.
The researchers found substantial linear structure across matched KV pairs. They designed a closed-form ridge mapper that selects the top-k most predictive source layers to generate the target's cache.
Experiments show this conversion runs 2.7 to 25x faster than reprocessing the context, which is crucial for optimizing cost-quality cascading, mid-conversation routing, and reducing LLM serving costs.
More from Infra
- Amazon Proposes Per-Request Validated Evidence Caching for Production RAG — _reachsumit · 2026-10-16
- LinkedIn's TransX report: sequence transduction for recsys, +6.02% CTR in A/B tests — _reachsumit · 2026-10-16
- llama.cpp PR boosts CPU inference speed with merged AVX2 tiled accumulators — jacek2023 · 2026-10-16
- Zero's next-gen IVM engine runs in a single DO, live-querying a 100GiB R2 database — threepointone · 2026-10-16
- Meme: driving to the chip factory in 2028 captures AI's compute anxiety — beffjezos · 2026-10-16
- One volunteer maintains the time zone file behind 4 billion phones — danbri · 2026-10-16