NVIDIA's Cross-Model KV Cache Transfer Speeds Up Inference by 25x
theomitsa · x · 2026-08-07
NVIDIA researchers introduced a cross-model KV cache transfer technique. When swapping between different-sized models in an LLM family (e.g., Qwen3 14B to 32B), this method allows the target model to reuse the source's KV cache, entirely skipping the prefill phase.
The researchers found substantial linear structure across matched KV pairs. They designed a closed-form ridge mapper that selects the top-k most predictive source layers to generate the target's cache.
Experiments show this conversion runs 2.7 to 25x faster than reprocessing the context, which is crucial for optimizing cost-quality cascading, mid-conversation routing, and reducing LLM serving costs.
More from Infra
- GPT-5.6 Price Cut Triggers Jevons Paradox: Token Consumption Jumps 10x — rohanpaul_ai · 2026-08-08
- Maestro v1.6.0 Adds MiniMax H3 Full Model Support and Local Video/Audio Generation — cocktailpeanut · 2026-08-08
- Altman Congratulates Oklo as Nuclear Reactor Achieves Criticality in Under a Year — sama · 2026-08-08
- Visualizing LLM API Price Volatility: An Open-Source Tool to Justify Local Compute Budgets — olddoglearnsnewtrick · 2026-08-08
- 10kAmp AI Chips Face Severe Power Delivery and Cooling Challenges — jwt0625 · 2026-08-08
- Three Practical Ways to Run Kimi K3 Locally Without Terabytes of RAM — theomitsa · 2026-08-08