2D-RoPE helps frontier language models copy long strings far more reliably
rohanpaul_ai · x · 2026-07-27
Frontier language models can handle advanced reasoning but still fail at a much simpler task: copying long, repetitive strings exactly.
The paper argues that standard 1D RoPE makes copying harder as sequences get longer because the needed token offset changes with length. It proposes 2D-RoPE, which lays text out as a 2D grid with row and column IDs so corresponding tokens align in the same column.
Key results:
- In synthetic copy tasks, one-layer models trained on lengths 1–100 copied perfectly at lengths up to 1,000× longer.
- The benefit also showed up in pretrained models up to 1.4B parameters.
- Common-sense performance stayed comparable at the tested scale.
The authors note the method still depends on line breaks, so it is not yet a universal fix.
Related event: 1D-RoPE Flaw Causes Frontier LLMs to Fail at Long String Copying(2 posts)→
More from Research
- Burkov skew AI hype: 'deterministic LLMs' and 'first agents' are old tricks rebranded — burkov · 2026-09-23
- Continuous diffusion beats discrete on random k-SAT, proposed as standard benchmark — ArashVahdat · 2026-09-23
- Grady Booch: Contemporary AI Still Lacks Abductive Reasoning, Just 'Next-Token Prediction' — Grady_Booch · 2026-09-23
- AI solves Navier-Stokes-related problem as machines upend mathematics, New Scientist reports — burny_tech · 2026-09-23
- Mathematician says OpenAI likely proved a significant partial case of the Hodge conjecture — burny_tech · 2026-09-23
- Code benchmarks are mostly slop: dev calls for narrow evals per domain, not one score — almmaasoglu · 2026-09-23