MIT's SwaT Transfers KV-Cache Across Model Families, Giving Text-Only LLMs Vision for Under 1 GPU-Day
alex_verem · x · 2026-11-12
MIT CSAIL and MIT-IBM Lab present SwaT (Swarm Telepathy), the first cross-family KV-cache transfer method: instead of text summaries, a translator converts one model's internal state into a form another vendor's model can consume, with no prefill. Qwen can hand memory to Llama, Gemma to Granite, with both models frozen.
Key results (12 open models, 6 families, 17 benchmarks):
- Receiving models keep >90% of their normal scores on average in every pairing
- With KV-cache compaction, 40% more accurate than passing a same-size text summary and 4x faster
- Translator trains in <1 GPU-day on <50M tokens
Surprise: trained on English text only, a text-only Llama receiving a Gemma's memory of an image averaged 39.6% on 5 visual QA tests vs 13.3% with a written caption, beating Gemma's own 34.2%; it also closed 3/4 of its gap on a Chinese knowledge test.
Caveats: each model pair needs its own translator, and the passed memory can carry information absent from any text — agents may communicate in a language nobody can audit.
Related event: MIT's SwaT Enables Cross-Family KV-Cache Transfer(2 posts)→
More from coding & agent
- Microsoft on agentic schedulers: LLM proposes, deterministic code validates — WirelessLife · 2026-11-12
- Visual block scripting drives an AI opponent in a classic arcade game — ezshine · 2026-11-12
- Gradient Labs' headless agents embed AI customer service inside banks' own stacks — neal_lathia · 2026-11-12
- Microsoft ships agent harness package for Agent Framework Go, easing agent setup — adnan_hashmi · 2026-11-12
- Dev releases open coding SFT dataset for Kimi K3.1 Pro after LFM3 1.7B tune's surprise popularity — alexcovo_eth · 2026-11-12
- fabric-planning-skill: open-source Copilot skill and Python CLI for Microsoft Fabric Plans — adnan_hashmi · 2026-11-12