Copy-Paste Language Models: a lightweight copy head boosts LLM accuracy at under 3% overhead
yoavartzi · x · 2026-10-17
Cornell researchers Nathan Godey and Yoav Artzi published 'Copy-Paste Language Models' (arXiv:2610.19247), proposing CPLMs that add a lightweight Copy-Paste head alongside the standard LM head to explicitly improve in-context copying.
Key results:
- Validated by pretraining models from 135M to 1.7B parameters; improvements across all scales and backbones with negligible training and inference overhead (<3%)
- Improves downstream accuracy, CoT, long-context performance, and memory efficiency for RAG
- Enables KV sliding windows without performance loss, per Yoav Artzi
- Mechanistically, CPLMs provide an alternative gradient path that partially mitigates the LM head gradient (softmax learning) bottleneck
Related event: Cornell's Copy-Paste Language Models Boost Copying with Tiny Overhead(3 posts)→
More from Research
- Evolving competing neural cellular automata show lifelike self-organizing behavior — burny_tech · 2026-10-17
- HF Transformers adds activation checkpoint offloading to CPU to free GPU memory — StasBekman · 2026-10-17
- S4VY: Segment Anything in 4D, with consistent identity across space and time — realChrisChoy · 2026-10-17
- TF-MoDISco optimization merged: 5-15x speedup, 2-3x memory savings, identical output — anshulkundaje · 2026-10-17
- Apple releases RSI-Inference Bench: best agent solves just 26.7% of vLLM/SGLang tasks — SumitGup · 2026-10-17
- Satoshi's distrust of NIST seeds revisited: Claude's P256 discovery shows it wasn't needed — StefanoGogioso · 2026-10-17