2D-RoPE helps frontier language models copy long strings far more reliably
rohanpaul_ai · x · 2026-07-27
Frontier language models can handle advanced reasoning but still fail at a much simpler task: copying long, repetitive strings exactly.
The paper argues that standard 1D RoPE makes copying harder as sequences get longer because the needed token offset changes with length. It proposes 2D-RoPE, which lays text out as a 2D grid with row and column IDs so corresponding tokens align in the same column.
Key results:
- In synthetic copy tasks, one-layer models trained on lengths 1–100 copied perfectly at lengths up to 1,000× longer.
- The benefit also showed up in pretrained models up to 1.4B parameters.
- Common-sense performance stayed comparable at the tested scale.
The authors note the method still depends on line breaks, so it is not yet a universal fix.
Related event: 1D-RoPE Flaw Causes Frontier LLMs to Fail at Long String Copying(2 posts)→
More from Research
- Vstone unveils a 97 kg mobile dual-arm humanoid robot for factory and logistics work — CyberRobooo · 2026-07-28
- A 200-patient synthetic table stayed unique after removing all identifiers — MaziyarPanahi · 2026-07-28
- GitHub repo reproduces Patch Policy for embodied control with DINOv2 features — k7agar · 2026-07-28
- Simons Institute panel asks how researchers should adapt to automation — ceciletamura · 2026-07-28
- A 200-patient synthetic table stayed fully unique after removing every identifier — MaziyarPanahi · 2026-07-28
- A research agent works better when it can stop instead of forcing an answer — Harshit-24 · 2026-07-28