Why LLMs Collapse Beyond Their Training Context: A Geometric Framework for RoPE
burkov · x · 2026-10-13
Modern LLMs rely on attention with Rotary Positional Embeddings (RoPE) to track word order, yet suffer severe performance collapse when input exceeds their original training context length. Prior work attributed this to rotational values drifting out of distribution, but the exact mechanical cause remained unexplained.
This article investigates why RoPE degrades over long inputs and establishes a unified geometric framework to explain and resolve the bottleneck.
More from Research
- Spatial biology slide costs stuck at $5-10k for five years, defying Moore's law — anshulkundaje · 2026-10-13
- Google Research trains user simulators with latent intrinsic feedback, cutting RL compute by over half — cocoweixu · 2026-10-13
- Helmholtz-SDE closes the approximation gap in simulation-free latent SDEs — scott_linderman · 2026-10-13
- Continual Learning Reading List: Cortical Labs Neurons, Self-Distillation, Text-to-LoRA — yacinelearning · 2026-10-13
- 1.5-Hour Deep Dive on Continual Learning: From First Principles to Qwen3-8B Experiments — yacinelearning · 2026-10-13
- Preference training may only rewrite final layers, with alignment and welfare implications — Sauers_ · 2026-10-13