HeteroFold transfers KV caches across model families with no receiver prefill, 10.7× faster at 32K

Prefill-Free Cross-Family KV Cache Transfer for Heterogeneous Multi-Agent LLMs

Vincent-Daniel Yun, Woosang Lim, Haneul Yoo, Sungjoo Yoo, Murali Annavaram, Sai Praneeth Karimireddy

cs.AI, cs.LG, cs.MA

2026-09-26

Maps a sender model's KV cache into a different family's receiver, skipping prefill: 10.7x faster at 32K and best cache-transfer results on all four long-context benchmarks.

What problem this solves

Heterogeneous multi-agent systems assign different model families to different roles, but agents still talk in text: every receiver must re-prefill the shared context the sender already processed and rebuild its own KV cache, and the duplicate cost grows with context length. Reusing the sender's cache is the obvious fix, but cross-family reuse has to clear three mismatches: tokenizers, layer counts, and KV representations. Existing prefill-free mapping methods (Dense Latent, KV Ridge) were validated only within a family; pushed cross-family they collapse. On Ministral-3-14B→Llama-3.1-8B, Dense Latent scores 2.72 F1 on Qasper against 44.64 for text communication.

There is a subtler trap. Low cache reconstruction error does not mean the receiver behaves correctly. In the paper's control, KV Ridge has lower reconstruction error than HeteroFold yet distorts receiver attention more. What must be preserved is the downstream computation the cache induces, not the cache values.

Method

HeteroFold freezes both models and learns fixed linear maps between them.

• Token Alignment (TA): pairs tokens across tokenizers by shared character-end boundaries. Receiver positions without an exact match reuse the sender state at the latest preceding shared boundary; many-to-one cases are not averaged. The ablation says this is the single biggest factor: swapping TA for same-index pairing drops GSM8K from 63.46 to 7.13.
• Layer Alignment (LA): proportional depth matching, each receiver layer fed three sender layers at offsets (−4, 0, +4). With a single sender layer, GSM8K falls to 46.17.
• Recolor: one affine map per receiver layer, separately for K and V. Sender features are whitened, aligned into the receiver's space with Procrustes alignment, then recolored to the receiver's mean and covariance. Keys are mapped before key normalization and RoPE, values after the projection; the receiver applies its own positional encoding.
• Output-aware calibration: Recolor aligns statistics but leaves attention-output error. A rank-16 correction trains separately for K and V: the key loss is attention-weight KL plus post-projection output error (where to attend), the value loss is the final attention-output error (what is retrieved). Only the low-rank factors train; they fold back into the affine maps, so inference adds one matrix multiply per layer and no module.

Calibration uses 1,600 training and 400 held-out prompts (Open-R1 problem text plus HotpotQA), no gold answers. One direction costs about two H100 GPU-hours and a 201–252M-parameter mapper, 0.4–0.5 GB in BF16.

Results

Six directions among Llama-3.1-8B, Qwen3-4B, and Ministral-3-14B. HeteroFold is the best cache-transfer method on all four long-context benchmarks in all six directions, and on most short-context ones. Ministral→Llama direction:

On QuALITY, 79.00 beats the text reference's 75.07; the other benchmarks in this direction still trail text. Giving both baselines the same TA lifts KV Ridge sharply, which confirms tokenizer mismatch as the main bottleneck, but HeteroFold stays ahead, so the gains are not all from TA. Asked to restate a question it can only see through the transferred cache (100 GSM8K prompts), the receiver reaches 83.0% ordered word overlap with HeteroFold, against 76.7% for KV Ridge + TA and 39.7% for Dense Latent + TA.

Latency at 32K context (Llama→Ministral, two H100s): native prefill takes 5,167 ms, HeteroFold 481 ms, a 10.74× speedup, and 1.18–1.47× faster than KV Ridge + TA (570 ms) and Dense Latent + TA (706 ms). On the multi-agent benchmark HiddenBench, average accuracy after 15 communication rounds is 30.6% against 29.2% for text, with a 3.2% invalid-answer rate (KV Ridge + TA hits 20.9%).

Why it matters

Teams building heterogeneous multi-agent systems can now price out repeated prefill: roughly 3.5–3.8× saved at 4K and 10.7× at 32K, with the saving growing as contexts get longer and shared more often. The methodological lesson travels further: evaluate cache transfer against receiver behavior (attention, outputs), because reconstruction error misleads. And since the learned corrections fold into plain affine maps with both models frozen, the serving path stays clean, which suits teams running multi-model serving on their own clusters.

The authors bill this as the first prefill-free cross-family transfer with explicit tokenizer alignment. Its honest positioning is opening the road: absolute quality still trails text communication, but the route is shown to work.

Limitations

• Clear quality gap versus text plus native prefill on most benchmarks (Qasper 33.72 vs 44.64, GSM8K 63.46 vs 85.67). The HiddenBench parity sits near 30% accuracy on 65 tasks, a sample size too small for strong claims.
• Each direction needs its own calibration: N models mean N(N−1) mappers, roughly two GPU-hours and 0.5 GB each, which stops scaling as the model pool grows.
• Evaluation covers three models from 4B to 14B on QA-style tasks only; larger models, code, and tool-calling agent workloads are untested.
• Both baselines were designed for same-family transfer and reimplemented here for cross-family use, so the comparison is unfavorable to them; the same-family appendix shows KV Ridge competitive, and the cross-family gaps deserve a careful reading.
• Memory is not free: with inter-GPU payload copies included, cache transfer does not always reduce peak receiver memory versus native prefill. If the sender has not processed the context yet, sender prefill adds about 3.4 s at 32K.

Terms

• prefill: the forward pass over the whole prompt before the first generated token; the dominant cost at long context.
• Procrustes alignment: classical method that finds the rotation best overlapping two paired point sets.
• whitening / recoloring: transforming features to unit, decorrelated variance, then restoring a target mean and covariance.
• GQA: grouped-query attention, where several query heads share one KV head.
• HiddenBench: multi-agent benchmark where each agent holds one private fact and must converse to recover the answer.

Source

• arXiv 2609.32259v2
• PDF
• Hugging Face
• GitHub · https://github.com/daniel-eai/Prefill-Free-Multi-Agent-LLMs

Related papers

• 273 idea atoms distilled from LLM papers drive a sampler toward directions no community would propose
• Meta Experts Match Vanilla MoE Loss at 70-80% Expert Budget
• Throwing away synthesized agent rules beats stored abstractions, 59.9% vs 49.0%
• Jev 1.13 lifts terminal-agent cost 19% on Terminal-Bench; pass rate stays 73%
• Synthetic questions cut replies 20% and lift OPD signal 28% over DAPO-17k
• Claude Code as meta-coach: hot-swapping user simulators and rewards mid-RL lifts success 36.7%

All paper explainers