Frontier LLMs still fail at faithful text copying
dair_ai · x · 2026-07-21
Key finding
Frontier LLMs can solve hard reasoning tasks, but they still fail at a deceptively simple ability: faithfully copying long text blocks.
Why it matters
The paper argues this is not just a trivial annoyance. Reliable copying underlies code reproduction, tables, structured documents, and many everyday agent workflows.
Proposed idea
The authors find that treating text as a 2D layout instead of a 1D token stream recovers much of the missing fidelity.
They introduce 2D-RoPE, which assigns each token a row and column ID so copying becomes a simpler retrieval problem.
Results
- In synthetic copy experiments, shallow Transformers with 2D-RoPE achieve perfect copying at lengths hundreds of times longer than seen in training.
- Standard positional encodings lag far behind.
- The advantage also holds in larger-scale pretraining on DCLM, up to 1.4B parameters.
Conclusion
The authors suggest that viewing text in 2D can improve language modeling, and that 2D positional encodings deserve further exploration.
More from Research
- Causal-only attention for non-generative tasks is wasteful, argues HF engineer — antoine_chaffin · 2026-09-11
- Catholic University of Chile researcher: scaling AI feedback is key to sustainable medical education — julianvarascom · 2026-09-11
- Nature paper images cellular activity across all organs, revealing body-wide circuits — arjunrajlab · 2026-09-11
- SignNet 1M Dataset Released for Sign Language Research — ducha_aiki · 2026-09-11
- ECCV26 Oral: Flow Matching Enables Single-Stage Multi-View Point Cloud Registration — ducha_aiki · 2026-09-11
- InFlux++ Method Released — ducha_aiki · 2026-09-11