Frontier LLMs still fail at faithful text copying
dair_ai · x · 2026-07-21
Key finding
Frontier LLMs can solve hard reasoning tasks, but they still fail at a deceptively simple ability: faithfully copying long text blocks.
Why it matters
The paper argues this is not just a trivial annoyance. Reliable copying underlies code reproduction, tables, structured documents, and many everyday agent workflows.
Proposed idea
The authors find that treating text as a 2D layout instead of a 1D token stream recovers much of the missing fidelity.
They introduce 2D-RoPE, which assigns each token a row and column ID so copying becomes a simpler retrieval problem.
Results
- In synthetic copy experiments, shallow Transformers with 2D-RoPE achieve perfect copying at lengths hundreds of times longer than seen in training.
- Standard positional encodings lag far behind.
- The advantage also holds in larger-scale pretraining on DCLM, up to 1.4B parameters.
Conclusion
The authors suggest that viewing text in 2D can improve language modeling, and that 2D positional encodings deserve further exploration.
More from Research
- New review: how prefrontal cortex dynamics turn world models into flexible behavior — AnnaCiaunica · 2026-10-27
- LIBERO-MAX: Robot Policies Drop 11–26 Points When the World Changes Mid-Task — drmapavone · 2026-10-27
- Kolab Benchmark Uses Pro Screen Recordings to Show AI Agents Overestimate Collaboration Quality — minnesotanlp · 2026-10-27
- One BBQ Example Pushes LLM Fairness Scores Past 90%, EMNLP Paper Questions Benchmarks — radamihalcea · 2026-10-27
- Real2Sem2Real turns one phone video into 2,000+ robot demos, matching 150 teleop demos — Scobleizer · 2026-10-27
- Understanding MuonH (Hyperball) through weight norm geometry of scale-invariant networks — jasondeanlee · 2026-10-27