AbridgeKV: latent KV cache compression matches full accuracy at 1/4 cache, 3.2× faster
DimitrisPapail · x · 2026-11-09
Inspired by the MEMENTO paper, this thread explores whether KV cache compaction needs explicit text at all. AbridgeKV trains the model to compress its own completed generation blocks into latent codes: K pause tokens read each finished block and write K code rows, while the raw rows are dropped.
- Keeps 1/4 of the generated KV cache while matching full-KV accuracy on a competition math suite (43.0 vs 43.8 avg@8)
- Finishes 3.2× faster than full KV on a single H100, far outperforming R-KV (0.32×) and LightThinker (1.4×)
- Resident KV memory grows far more slowly than full KV over a 16K-token generation
The result shows latent compaction is viable for long reasoning without any textual summaries.
More from Infra
- macOS 27.3 lets Mac VMs use host GPU and Neural Engine, up to 7× faster Core AI — SumitGup · 2026-11-09
- Developer builds Shura: a local coding assistant on one RX 7800 XT hitting 78 tok/s that should improve with use — HIKKIAN · 2026-11-09
- Altman says home DRAM is a waste as memory prices soar; critics blame OpenAI's datacenter buildout — jonerp · 2026-11-09
- DeepSeek-V4 in 64GB RAM decodes at just 9-15 tok/s — a capacity trick, not an inference win — petrusenko_max · 2026-11-09
- Public GitHub pushes double in one year to 408M per quarter amid AI-generated code surge — DimitrisPapail · 2026-11-09
- McKinsey: multi-step agents burn 5-30x more tokens than chat — ingliguori · 2026-11-09