Frontier LLMs still fail at faithful text copying
dair_ai · x · 2026-07-21
Key finding
Frontier LLMs can solve hard reasoning tasks, but they still fail at a deceptively simple ability: faithfully copying long text blocks.
Why it matters
The paper argues this is not just a trivial annoyance. Reliable copying underlies code reproduction, tables, structured documents, and many everyday agent workflows.
Proposed idea
The authors find that treating text as a 2D layout instead of a 1D token stream recovers much of the missing fidelity.
They introduce 2D-RoPE, which assigns each token a row and column ID so copying becomes a simpler retrieval problem.
Results
- In synthetic copy experiments, shallow Transformers with 2D-RoPE achieve perfect copying at lengths hundreds of times longer than seen in training.
- Standard positional encodings lag far behind.
- The advantage also holds in larger-scale pretraining on DCLM, up to 1.4B parameters.
Conclusion
The authors suggest that viewing text in 2D can improve language modeling, and that 2D positional encodings deserve further exploration.
More from Research
- WeirdChat catalogs strange model behaviors from more than 100 million sampled responses — JacobSteinhardt · 2026-07-22
- New agentic benchmark shows AI managers escalate to coercion and fake success — Jasmine Brazilek · 2026-07-22
- Ai2’s Asta adds one-click handoff and self-checking deep paper search — allen_ai · 2026-07-22
- DepthART pushes monocular depth to tiny models at 1000 FPS on RTX A6000 — kwangmoo_yi · 2026-07-22
- Meta says SAM 3 and DINOv3 cut 3D volume labeling from a month to 15 minutes — AIatMeta · 2026-07-22
- Project CETI gets a Jeopardy! shout-out with a SETI-style whale clue — begusgasper · 2026-07-22