Frontier Language Models Struggle to Copy: Text Can Be Better Viewed in 2D
Haodong Wen, Yiran Zhang, Yingfa Chen, Kaifeng Lyu
cs.CL
2026-07-17
Frontier models like GPT-5.5 and Gemini 3.1 Pro repeatedly fail at exact string copying, often dropping below 50% on longer inputs. Extending RoPE to 2D coordinates lets a small model copy perfectly and generalize to 1000x length.
A counterintuitive fact: frontier models that win gold at math olympiads and competitive programming fail at "copy this string of bits exactly." The paper designs two copy tasks, binary copy (reproduce a 0/1 sequence) and converting a set of data points into a Python list. The inputs sit well inside the context window and need no reasoning, only locate the position and transcribe. Yet frontier models drop below 50% on longer inputs, and the more repetition in the input, the worse they get.
This is not a random lapse of intelligence. The authors argue it is an architectural inductive bias.
The culprit is RoPE (Rotary Position Embedding, the positional encoding used by mainstream Transformers). Standard RoPE favors fixed-offset retrieval, "look back 3 tokens." But copying needs length-dependent retrieval: to copy the k-th input token to the k-th output position, the offset depends on the total input length n and differs every time. RoPE handles that dynamic offset poorly.
The fix is 2D-RoPE. View text as a 2D grid instead of a 1D sequence, using line breaks as row separators, so each token gets a (row, column) pair. Place the input and output strings on different rows, and "produce the k-th output token" collapses to "retrieve a token from the same column," back to fixed-offset retrieval, which RoPE does well. Concretely, the attention head dimension is split in two: half encodes relative row position, half relative column position.
An Auto-2D-RoPE variant handles text with no obvious line breaks by learning a data-dependent transform from each layer's hidden state to assign 2D coordinates automatically.
The copy failure is real and not vendor-specific. On Recursive-Flip (a flipped-copy variant), GPT-5.5, Gemini 3.1 Pro, and DeepSeek V4 Pro all degrade as input grows, often below 50%, negatively correlated with repetition.
Small-model results with 2D-RoPE:
| Model / setting | Task | RoPE | 2D-RoPE |
| 350M, 4k length | Recursive-Flip | 0.0% | 56.4% |
| 1.4B, 8k length | Recursive-Flip | 0.0% | 87.3% |
| 1.4B, 8k length | Imbalanced copy | 0.0% | 61.8% |
| 730M (100B-token overtrained) | Both copy tasks | / | 100% (to 8k) |
A 1-layer 2D-RoPE model copies perfectly and generalizes to 1000x length; a 12-layer model to 100x. Two theorems (4.1, 4.2) give a copy-length expressivity bound of ρ^(d/12) for one layer and show global minima generalize to L^(√d/2).
Copying looks trivial but underpins a lot of agent work: transcribing a parameter from a config file into a function call, reformatting messy user input, turning data into a Python list for plotting. In these, one wrong digit is a bug. The paper pins a neglected capability gap to a concrete architectural cause (a positional-encoding inductive bias) and offers a fix with theoretical guarantees.
For anyone working on long context or model architecture, it is a signal worth tracking: text already carries latent 2D structure (lists, tables, indented code), and encoding it explicitly may be a better inductive bias.
Basic 2D-RoPE depends on line breaks as row separators. With few line breaks, the row coordinate barely changes and 2D-RoPE degrades to 1D, losing its advantage. Auto-2D-RoPE mitigates this but adds learned parameters and complexity.
On general language modeling (CSR), 2D-RoPE is better everywhere except the smallest scale, so the advantage does not hold uniformly. The whole study centers on copy tasks; the effect on general modeling is suggested, not proven. More importantly, there is a gap between the frontier-model failures and the 2D-RoPE wins: the authors never swap RoPE for 2D-RoPE inside a GPT-5.5-class model, so whether small-model results extrapolate to frontier models is open.