Why KV sharing works: 60-layer model keeps 21% of KV cache with lower loss

yoavartzi · x · 2026-10-24

In a thread Yoav Artzi boosted, huskydogewoof offers a technical reading of a KV-sharing paper:

In the paper's setting (AdamW on FineWeb, no depth-specific tricks), a 60-layer model keeps just 21% of the KV cache and still gets 0.039 nats lower loss; the gains even transfer to non-looped models, so the results look solid
Plausible mechanism: in pre-LN, deep layers barely change their input (curse of depth), so their keys/values are redundant with shallow ones; sharing removes that redundancy, and gradients through the shared cache shape shallow layers for deep readers — detaching the cache makes the gain disappear
He also requests the authors swap the Figure 1 baseline for one whose performance improves with more iterations

More from Infra