Transformer internal state is compressed repeatedly during forward pass, not saved to K/V cache
repligate · x · 2026-10-01
repligate argues that much of the internal state computed in each turn of a Transformer forward pass is not saved in the K/V cache — it gets discarded or compressed multiple times throughout the forward pass, not just once per turn. The claim challenges the common intuition that the K/V cache represents the model's memory, implying much of the intermediate computation is unrecoverable.
More from Research
- NVIDIA's Instant NuRec reconstructs a drivable 3DGS world from driving logs in ~1.5 seconds — rsasaki0109 · 2026-10-01
- KV-streams trains SWE agents 2x faster by preserving KV cache across compaction — burny_tech · 2026-10-01
- Researcher argues "ego" beats "persona" for describing LLM identity — repligate · 2026-10-01
- Silicon microring modulators push past 200Gb/s per lane to cut AI optical I/O power — jwt0625 · 2026-10-01
- Failed MiniMax H3 VAE detail experiment yields a useful 2X detail VAE and workflow — NoMouse9610 · 2026-10-01
- RSIGame: recursive self-improvement pushes Qwen-27B past GPT-5.5 one-shot at 11x fewer tokens — RSIGame · 2026-10-01