Looped CED cuts KV-cache memory 36.8x on 552B model, but encoder loops cost compute
teortaxesTex · x · 2026-11-21
The paper "Looping Deeper Without Growing the KV Cache" introduces Looped CED, which repeatedly processes the encoder layers that generate the global KV cache. Training with varying loop counts makes representations converge, so only the final loop's KV cache needs storing. Serving tests with a 552B DeepSeek model at 8 loops show 36.8x KV-cache memory reduction and up to 3.6x throughput gain.
teortaxesTex adds caveats from the paper's figures: encoder-loop bpb gains grow with scale (1.3–1.6x effective parameters; each looped pass worth 1/4–1/8 of a real layer), but inference evaluates the full loop stack each round — naively applied to V4.1-Flash that's 147 layers, 7.4x slower than normal operation. The authors themselves note the 552B model was never trained to loop, so extrapolation isn't supported.
More from Infra
- Microsoft launches MAI Licensed Weights; Purdue researcher asks to run them on own hardware — marlene_zw · 2026-11-21
- Agents are sessions, not requests: Google explains sizing GKE node pools for agent workloads — rseroter · 2026-11-21
- AI and chip stocks plunge: Micron -12.6%, Arm, CoreWeave, Anthropic all down over 10% — xiaohu · 2026-11-21
- Ben Bajarin: AI is the deployment phase of the Information Age, investors unconvinced — BenBajarin · 2026-11-21
- 4 essential reads to understand LLM decode from first principles — CatAstro_Piyush · 2026-11-21
- High-Flyer did try to obtain 75K GPUs legally, per teortaxesTex — teortaxesTex · 2026-11-21