Is RoPE Just a Recency Bias? Experiments Show It's Nearly Removable Late in Training
BlackHC · x · 2026-11-03
BlackHC (Andreas Kirsch) published a deep dive asking whether RoPE is mostly a recency bias—the intuition being that rotating distant keys turns matches into noise, biasing attention toward nearby tokens.
Key findings:
- Partly right: rotation does wash out content matches with distance. But trained models do more: they concentrate content in the slowest-frequency pairs (which barely rotate) and use the fastest frequencies to build exact-offset heads (e.g., previous-token heads), which only require a fixed rotation inside the Q/K projections—no MLP needed.
- Barbero et al. (ICLR 2025) found Gemma 7B prefers RoPE's lowest frequencies (suspected semantic carriers) and uses the highest frequencies for positional attention patterns.
- p-RoPE ablations: unrotating the slowest 25% of pairs in Gemma 2B yields Wiki ppl 4.44 vs 4.46 for full RoPE; unrotating 75% hurts (4.53); NoPE is 4.86.
- DroPE (ICLR 2026) drops RoPE mid-run in a 500M model: full RoPE 21.72 ppl; dropped at step 14K/16K: 21.73 (nearly lossless); dropped at 8K: 22.42; NoPE from scratch: 23.77. RoPE matters mostly during training and becomes dispensable at the end (in-domain).
- Long-context implication: at base 10K only 9 of 64 pairs stay coherent across 4K tokens, none at 32K—explaining Llama 3's base of 500K. Rescaling ωᵢ perturbs offset heads: Position Interpolation disrupts them, YaRN spares the fast pairs.
Open question: whether the training benefit comes from exact-offset heads or mere recency—at 120M params, ALiBi matches RoPE's perplexity. The author notes simulations and figures were produced with Claude and Gemini, then edited by him.
More from Models
- Perplexity distills 9B embedding teacher into 0.6B model sharing the same embedding space — antoine_chaffin · 2026-11-03
- Astra's Qwen weight edit was a published interpretability technique, not RSI — markjeffrey · 2026-11-03
- Yoav Goldberg: Astra's weight-editing feats are mostly known interpretability tricks — yoavgo · 2026-11-03
- Independent analyses with Claude and Grok suggest Qwen3.5-27B was grown from the 9B checkpoint — andrewgwils · 2026-11-03
- GPT-6 Astra edits just 0.016% of a small model's weights for big math gains — kevrussell · 2026-11-03
- Perplexity releases open-weight contextual embedding models (9B/0.6B), sets SOTA on ConTEB — AravSrinivas · 2026-11-03