Is RoPE just recency bias? Visual analysis shows distant matches dephase into noise
BlackHC · x · 2026-11-03
Andreas Kirsch (BlackHC) asks whether RoPE is essentially a recency bias:
- His intuition: rotating a distant key turns its match with the query into "noise," so attention initially favors nearby tokens; attending to an exact dynamic offset would need an MLP to undo the rotation — seemingly hard.
- An accompanying visualization shows 64 frequency pairs rotating at different speeds, with distant matches decohering into noise; slow pairs carry content while fast pairs aim exact-offset heads; the sum of all pairs is recency-dominated.
- Verdict: "partly right," plus a conjecture that RoPE might be droppable late in training — maybe a scaffold.
Drafted with Claude & Gemini Argon; blog at blog.blackhc.net.
More from Research
- Zyphra Introduces Hierarchical Attention, Claiming up to 400x Speedups at 1B-Token Contexts — teortaxesTex · 2026-11-03
- Open-source sparse attention: Qwen 27B with no accuracy loss, 10M-token context — Immediate_Sky_6566 · 2026-11-03
- Commentary: Zyphra's ZHA Is Principled and 'Blows Secret Subquadratic Labs Out of the Water' — teortaxesTex · 2026-11-03
- Perplexity distills 9B embedding teacher into 0.6B model sharing the same embedding space — antoine_chaffin · 2026-11-03
- Prior contextual embedding training treats supporting-context chunks as negatives — antoine_chaffin · 2026-11-03
- Astra's Qwen weight edit was a published interpretability technique, not RSI — markjeffrey · 2026-11-03