Nvidia Paper: Cross-Model KV Cache Reuse Speeds Up Inference by 25x

rohanpaul_ai · x · 2026-08-07

A new Nvidia paper introduces a method to reuse prompt memory (KV Cache) across different LLMs. Typically, switching models requires the new model to re-read the entire conversation to build its cache. This approach uses a simple linear converter to let related models reuse cached memory, skipping the reprocessing of long conversations.

Technical Details & Performance:

The mapper learns how target attention heads relate to source layers, removing and restoring position info before and after conversion.
It learns from just 500 calibration sequences without backpropagation, requiring matching cache shapes.
Across 6 model pairs (Qwen3, Llama 3.1, etc.), it retained 73% to 98% of the target model's benchmark accuracy.
Conversion ran 2.7 to 25 times faster, with Qwen3 handoffs remaining stable across 10 turns.

The research notes that success depends less on total conversion error and more on whether errors land where attention actually reads.

Original post →

Nvidia Paper: Cross-Model KV Cache Reuse Speeds Up Inference by 25x

More from Infra