BeNice! KV-sharing for looped Transformers cuts GPU memory 66%, TTFT latency 83.6%
realDanFu · x · 2026-10-26
A paper from UC San Diego and Together AI (including Daniel Fu) proposes BeNice!, tackling the problem that looped Transformers generate independent KV-caches per recurrence, making memory grow linearly.
The two-stage method first applies stochastic-depth pretraining to induce contractive residual dynamics, pushing representations across recurrences toward a common limit; then BeNice! post-training broadcasts the final recurrence's KV-cache into the first recurrence, training compatible residuals while preserving quality.
On a 1.3B model: total GPU memory drops 66.2%; in simulated long-context workloads throughput improves up to 23.6% and 90th-percentile time-to-first-token drops 83.6% with no quality loss. KV-sharing composes with memory-efficient mixers (MLA, Kimi delta attention) and low-precision KV quantization (fp8, mxfp8, nvfp4), improving cache hit rates by up to 31.4%.
More from Infra
- Anthropic engineer: Claude Code cache hit rates are now high, savings live — teortaxesTex · 2026-10-26
- Six-month bet: hybrid AI deployments, not a wholesale shift to local models — yi111 · 2026-10-26
- Deco decouples 'thinking' from 'spelling' in generative recommenders for 3.2-8.6x throughput — _reachsumit · 2026-10-26
- Redditor with $4,000-$5,000 budget seeks local AI video generation hardware advice — dewsphere · 2026-10-26
- ThousandML Open-Sources CQ 4-bit QLoRA: 17.4% Lower Val Loss Than Unsloth NF4 — textclf · 2026-10-26
- Why OpenAI and Anthropic Cut LLM Prices by Up to 80% in Months — Azer_pubes33 · 2026-10-26