CPLM: sliding-window Transformer matches full attention with 12x less KV cache
burny_tech · x · 2026-10-21
Cornell's Nathan Godey and Yoav Artzi propose Copy-Paste Language Models (CPLM), adding a lightweight Copy-Paste head on top of the standard LM head to explicitly model in-context copying. A 256-token sliding-window CPLM matches full-attention perplexity on FinePDF with 12x less KV cache memory and jumps from 14.3% to 99.4% on 64k needle retrieval. Validated from 135M to 1.7B parameters with negligible overhead. The mechanism: 60% of pretraining tokens repeat within a 4k window, but the low-rank LM head makes copying hard to learn, creating a hidden gradient bottleneck that the CP head partially mitigates via an alternative gradient path.
More from Research
- TokenGate: learned token pruning speeds up LLM prefill up to 11.3x — Liana Patel · 2026-10-21
- Terence Tao proposes AI-assisted challenge to revisit a classical Gauss problem — robleclerc · 2026-10-21
- OSWorld-Science benchmark: 12 VLMs average just 34.3% on real scientific software tasks — jiqizhixin · 2026-10-21
- Experiments show Qwen3 can compile arbitrary patterns into layer-by-layer steps with plan training — Sauers_ · 2026-10-21
- 'Functional Welfare' paper shows repeated rejections degrade Gemma 3 27B behavior — repligate · 2026-10-21
- Someone actually trained a model with zig-zag activations, code and weights open-sourced — Sauers_ · 2026-10-21