Looped CED cuts KV-cache memory 36.8x on 552B model, but encoder loops cost compute

teortaxesTex · x · 2026-11-21

The paper "Looping Deeper Without Growing the KV Cache" introduces Looped CED, which repeatedly processes the encoder layers that generate the global KV cache. Training with varying loop counts makes representations converge, so only the final loop's KV cache needs storing. Serving tests with a 552B DeepSeek model at 8 loops show 36.8x KV-cache memory reduction and up to 3.6x throughput gain.

teortaxesTex adds caveats from the paper's figures: encoder-loop bpb gains grow with scale (1.3–1.6x effective parameters; each looped pass worth 1/4–1/8 of a real layer), but inference evaluates the full loop stack each round — naively applied to V4.1-Flash that's 147 layers, 7.4x slower than normal operation. The authors themselves note the 552B model was never trained to loop, so extrapolation isn't supported.

Original post →

More from Infra

Infra channel →