Tsinghua's Latent Compaction Compresses KV Cache 32× While Keeping 96.7% Retrieval Accuracy
Yingfa Chen · hf · 2026-10-15
Tsinghua NLP introduces Latent Compaction, which learns a compact document-specific KV cache via test-time training: cache entries are initialized using reconstruction importance, teacher attention, and key-space diversity, then jointly optimized by end-to-end distillation from the full-cache model with weights frozen and no access to downstream questions. On RULER it achieves 96.7% accuracy at 32× compression vs 40.0% for KVZip+, with gains on LongBench and Phonebook Lookup and transfer across Qwen3 and MiniCPM5. Code released.
More from Infra
- Gumroad cut $100k/year from AWS by moving one Rails webhook endpoint to Go on 3 servers — DanielLockyer · 2026-10-15
- Anthropic's margins imply 70-75% of its compute goes to R&D, analysis shows — Jsevillamol · 2026-10-15
- First PyTorch Madrid meetup demos llm-d: K8s-native distributed LLM inference on vLLM — SergioPaniego · 2026-10-15
- Starlink Wi-Fi spotted on Austrian Airlines, speedtest shows 157 Mbps down at 43ms ping — mitsuhiko · 2026-10-15
- Banks try to offload AI data center debt they just underwrote — jazir55 · 2026-10-15
- ContinuityBench: Stateful Failover Preserves LLM Conversations at 99.2% Across Providers — its_vayishu · 2026-10-15