Is RoPE just a recency bias? Rotation turns distant keys into noise
BlackHC · x · 2026-11-03
BlackHC examines the intuition that RoPE is essentially a recency bias: rotating a distant key turns its match with the query into "noise", so attention initially favors nearby tokens; attending to an exact dynamic offset would require an MLP to undo the rotation, which seems hard. He says this is partly right, and explains the dephasing mechanism: for q = k each dimension pair adds cos(ωᵢΔ), and pairs rotating at different speeds drift out of phase until the match sits at chance, the ±1/√d cosine of an unrelated key. The fastest 16 pairs die after 4 tokens, the middle 32 after 512, the slowest 16 after 4K.
More from Research
- Zyphra Introduces Hierarchical Attention, Claiming up to 400x Speedups at 1B-Token Contexts — teortaxesTex · 2026-11-03
- Commentary: Zyphra's ZHA Is Principled and 'Blows Secret Subquadratic Labs Out of the Water' — teortaxesTex · 2026-11-03
- Perplexity distills 9B embedding teacher into 0.6B model sharing the same embedding space — antoine_chaffin · 2026-11-03
- Prior contextual embedding training treats supporting-context chunks as negatives — antoine_chaffin · 2026-11-03
- Astra's Qwen weight edit was a published interpretability technique, not RSI — markjeffrey · 2026-11-03
- Yoav Goldberg: Astra's weight-editing feats are mostly known interpretability tricks — yoavgo · 2026-11-03