Is RoPE just a recency bias? Analysis suggests long-range matching degrades to noise

Andreas Kirsch (BlackHC) analyzed whether RoPE mainly acts as a recency bias: rotating distant keys turns their matches into noise, with only 9 dimension pairs in Gemma surviving beyond 4K tokens, and RoPE nearly losslessly removable late in pretraining.

2026-11-03 ~ 2026-11-03 · 4 related posts

• Is RoPE just recency bias? Visual analysis shows distant matches dephase into noise — BlackHC · 2026-11-03
• Is RoPE just a recency bias? Rotation turns distant keys into noise — BlackHC · 2026-11-03
• How RoPE rotation kills long-range matching: only 9 of 64 dims survive 4K tokens — BlackHC · 2026-11-03
• Is RoPE Just a Recency Bias? Experiments Show It's Nearly Removable Late in Training — BlackHC · 2026-11-03