RoPE and ALiBi can be seen as KDA variants with imaginary or real log decay
teortaxesTex · x · 2026-07-28
A short technical claim ties several position-encoding ideas together:
- RoPE is framed as equivalent to KDA with imaginary log-decay factors.
- ALiBi is described as KDA with real log-decay factors.
- State-space models are said to not need explicit positional embeddings because position is already baked into the recurrence.
The post is really a compact theory note about how different sequence models encode position, and it points toward a unified view rather than isolated tricks.
Related event: Kimi K3 Architecture Drops RoPE for NoPE(3 posts)→
More from Research
- Yale PhD student open-sources his paper figure scripts, packaged as a Skill for Claude Code and Cursor — burny_tech · 2026-09-23
- AI models now match superforecasters on ForecastBench; rematch set for October — burny_tech · 2026-09-23
- Dev uses Opus 5.5 with Lean to formally verify Claude Agent SDK, yielding 16 bug-fix PRs — bcherny · 2026-09-23
- Mathematicians, not just LLMs, made AI's math breakthroughs possible, scholars argue — tak3sh8 · 2026-09-23
- AI-enabled drug discovery cuts discovery time by 15-80%, McKinsey research finds — menhguin · 2026-09-23
- Gemini training details dissected: groupwise reward redistribution to fight reward hacking — nrehiew_ · 2026-09-23