Cornell's Copy-Paste LMs add a lightweight head to boost in-context copying and long-context retrieval
lil-lab · hf · 2026-10-16
Cornell researchers Nathan Godey and Yoav Artzi propose Copy-Paste Language Models (CPLMs), adding a lightweight Copy-Paste head to the standard LM head to explicitly model in-context copying.
- Validated by pretraining models from 135M to 1.7B parameters, with improvements across all scales and backbones at negligible training and inference overhead
- Improves training efficiency, downstream performance, and offers new operating points for long-context and retrieval settings
- The authors attribute the gains to an alternative gradient path that partially mitigates the LM head gradient bottleneck
More from Research
- Tsinghua professor's ApexIntelligence raises $50M to build self-evolving foundation models — DeryaTR_ · 2026-10-16
- Anders Sandberg on fluid computation: error budgets tighten quadratically over time — anderssandberg · 2026-10-16
- How the fluid computation works: force field advects particle to 0/1 boxes — anderssandberg · 2026-10-16
- FlashAttention creator Tri Dao joins Radical Numerics as scientific advisor — exnx · 2026-10-16
- SAIR Launches Open Math OS, Offering Free Compute and 100+ Tools for Community-Owned Math AI — Thom_Wolf · 2026-10-16
- JapanFold Claims One Tenstorrent Server Beats an Nvidia DGX B200 on Bio Models at a Third of the Price — DavidBennett__ · 2026-10-16