How RoPE rotation kills long-range matching: only 9 of 64 dims survive 4K tokens
BlackHC · x · 2026-11-03
BlackHC draws on Barbero et al. to explain long-range matching under RoPE: for q = k, each dimension pair contributes cos(ωᵢΔ), and since pairs rotate at different speeds, the terms drift out of phase until the match decays to chance level (±1/√d).
Concretely: the fastest 16 pairs dephase after 4 tokens, the middle 32 after 512, and the slowest 16 after 4K. With a base of 10K, only 9 of 64 pairs stay under a quarter turn across 4K tokens, and none at 32K — explaining Llama 3's base of 500K. He also notes Gemma 7B concentrates most of its Q/K norm in the slowest pairs.
More from Research
- Zyphra Introduces Hierarchical Attention, Claiming up to 400x Speedups at 1B-Token Contexts — teortaxesTex · 2026-11-03
- Commentary: Zyphra's ZHA Is Principled and 'Blows Secret Subquadratic Labs Out of the Water' — teortaxesTex · 2026-11-03
- Perplexity distills 9B embedding teacher into 0.6B model sharing the same embedding space — antoine_chaffin · 2026-11-03
- Prior contextual embedding training treats supporting-context chunks as negatives — antoine_chaffin · 2026-11-03
- Astra's Qwen weight edit was a published interpretability technique, not RSI — markjeffrey · 2026-11-03
- Yoav Goldberg: Astra's weight-editing feats are mostly known interpretability tricks — yoavgo · 2026-11-03