Cache-to-Cache: Direct Semantic Communication Between Large Language Models
Tianyu Fu, Zihan Min, Hanling Zhang, Jichao Yan, Guohao Dai, Wanli Ouyang, Yu Wang
ICLR'26
cs.CL, cs.LG
2025-10-04
C2C fuses a Sharer's KV cache into a Receiver with residual gating. It beats solo models by 6.4–14.2 points and text handoff by 3.1–5.4, with 2.5× lower latency.
Multi-LLM systems still pass notes in text. One model squeezes a high-dimensional KV cache into a token string; the other has to unpack it. Structure dies in that round trip. The paper's running example: a Coder tells a Writer to insert at <p>, the Writer treats <p> as a generic tag, and the paragraph lands in the wrong place.
Text is also slow and vague. Every handoff waits on sequential decoding. Protocols such as MCP and A2A standardize the envelope, not the meaning. The experiment is whether models can skip the string and share internal state.
Two oracles come first. Cache enrichment: prefill on few-shot exemplars plus the question, drop the exemplar slice, decode from a question-length cache. On Qwen3 this Oracle scores 62.34% against Direct 58.42% and full few-shot 63.39%. The extra accuracy lives in how the question is encoded, not in extra tokens to attend over. Layer by layer the picture splits: a few layers gain, most lose, which is why the fuser later learns a gate.
Cache transformation: a 3-layer MLP maps Qwen3-4B KV into Qwen3-0.6B space. t-SNE puts the raw caches far apart; after the map they sit inside the target cloud, covering only a subset. Correct-answer sets of different models overlap only partly. Complementary encodings are real.
C2C names the donor Sharer and the generator Receiver. At prefill both models write KV. A fuser projects the Sharer's cache into the matching Receiver layer and adds it back as a residual. Three parts: concat-then-project plus feature fusion; input-aware head reweighting; a per-layer Gumbel-sigmoid gate that hardens to binary at inference. Both LLMs stay frozen. Only the fuser trains, with next-token prediction on the Receiver given the fused cache. The main run uses the first 500k OpenHermes-2.5 samples for one epoch.
Alignment is mechanical. Tokens: decode each Receiver token to a string, re-encode with the Sharer tokenizer, keep the longest covering piece on one-to-many maps. Layers: terminal alignment, last layer to last layer, walking backward to the shallower model's first layer.
The Receiver is fixed as Qwen3-0.6B. Three Sharers: Qwen2.5-0.5B, Llama3.2-1B, Qwen3-4B-Base. Four multiple-choice suites: MMLU-Redux, OpenBookQA, ARC-Challenge, C-Eval. Zero-shot, greedy, 64-token cap.
| Sharer | vs Receiver | vs T2T | speedup vs T2T |
| Qwen2.5-0.5B | +11.00 | +5.36 | 3.46× |
| Llama3.2-1B | +9.64 | +4.15 | 1.51× |
| Qwen3-4B-Base | +11.88 | +3.06 | 14.41× |
Qwen3-4B-Base barely follows instructions (1.03 on MMLU-Redux). T2T still waits on its analysis, up to 7.54s. C2C never asks it to speak, and the instruction-tuned small Receiver still absorbs the base model's knowledge.
Latency on MMLU-Redux: the T2T Sharer decodes about 80 communication tokens in 1312ms; C2C replaces that with 90ms of parallel fusion. The abstract's 2.5× average mainly excludes this pathological Base pairing. The 6.4–14.2 point range versus "individual models" mixes Sharer and Receiver depending on the pair.
Ablation is blunt. Overwriting the Receiver cache with a projected Sharer cache scores 20.70 average, worse than the Receiver alone. Residual fusion jumps to 44.88; the gate adds 3.07 to 47.95. Same-model C2C (Sharer = Receiver = Qwen3-0.6B, 529M trainable) already beats full SFT of the Receiver (596M). Heterogeneous C2C is higher still, with a 478M fuser. The gain is not extra capacity, but the fuser is close to the small model's size, not a LoRA-scale add-on.
Cross-family holds. Gemma3-1B, Qwen2.5-Math-1.5B, and Qwen2.5-Coder-0.5B as Sharers all beat T2T on MMLU-Redux; five pairings average +8.59 points. Swap Qwen2.5-0.5B and Qwen3-0.6B roles: C2C still +5.05, T2T falls 6.30. On LongBench, T2T at 8k+ drops below the Receiver (25.64 vs 25.99); C2C stays at 30.72.
A heavier C2C-C variant first maps Sharer cache through a 3-layer MLP. With Qwen3-4B into 0.6B it recovers 76–86% of the teacher-student gap. On GSM8K a lone Qwen3-0.6B scores 41.17, a T2T agent flow 61.18, plain C2C 62.55, and text interpretation plus C2C 78.01. 300 training steps (under 9 GPU hours) already sit near the final checkpoint.
The shift is from mailing letters to sharing working memory. Routing picks one model; C2C uses both encodings at once. Anyone already cascading models, running speculative decoding, or handing a large model's context to a small decoder already holds KV caches. C2C spends those tensors instead of another text round.
The practical cost is a fuser per pair. The direction is clear. The step is incremental.
The authors list two. A weak Sharer pollutes a strong Receiver; T2T and C2C both eat that. An accounting item that Qwen3-0.6B alone answers A becomes B once Qwen2.5-0.5B injects a wrong reading. Scaling past pairs still means pairwise training; an O(N) shared latent space is only sketched in the appendix.
The eval is narrow. Main tables are multiple choice, 64 tokens, temperature 0. Open-ended generation, tool use, and real agent loops get one GSM8K toy. Fusion hits the input prefill cache; the generated prefix still uses the Receiver's own KV, so thought is not fully shared. C2C-C looks stronger, but the main paper sticks to the simple fuser, and generation length is not matched across those tables.