Cornell/IBM: looped transformers can share KV-cache across recursions, cutting memory 76-79% losslessly
WeijiaShi2 · x · 2026-10-24
Yoav Artzi's team presents 'The Surprising Effectiveness of Shared Memory in Looped Transformers' (BeNice): looped LMs add compute without more parameters, but each recursion writes its own KV-cache so memory still grows. Pretraining the model to write the cache only on the first recursion and share it (with a short window) turns out lossless — and even improves quality. At 150M-1B params, their LPT and hybrid variant set a new quality-memory frontier: with five recursions the hybrid cuts relative validation perplexity by 1.12-1.82 on FineWeb-Edu vs a same-size standard transformer while using 76-79% less context memory. Shared and local memory develop different representations; later recursions attend mostly to shared memory, which also acts as a gradient highway.
More from Infra
- ai& cuts MiniMax H3 video generation from 229s to 19s across NVIDIA, AMD, Tenstorrent — DavidBennett__ · 2026-10-24
- Local AI benchmarks miss thermal throttling — a stand can boost NVIDIA Spark perf — draginol · 2026-10-24
- Cloudy: autonomous cloud research agent hand-writes Triton kernels that beat tuned vLLM by up to 15% — saurabhtwq · 2026-10-24
- Chamath: major US bank cut AWS and GCP spend, built six self-run data centers — SumitGup · 2026-10-24
- OpenRouter slide shows token usage exploding as blended cost per token keeps falling — aniketmaurya · 2026-10-24
- Jack Dorsey touts full-duplex realtime voice model trained on a single GB200 NVL72 — TinfoilTricorn · 2026-10-24