RoPE and ALiBi can be seen as KDA variants with imaginary or real log decay
teortaxesTex · x · 2026-07-28
A short technical claim ties several position-encoding ideas together:
- RoPE is framed as equivalent to KDA with imaginary log-decay factors.
- ALiBi is described as KDA with real log-decay factors.
- State-space models are said to not need explicit positional embeddings because position is already baked into the recurrence.
The post is really a compact theory note about how different sequence models encode position, and it points toward a unified view rather than isolated tricks.
Related event: Kimi K3 Architecture Drops RoPE for NoPE(3 posts)→
More from Research
- OpenRoboto launches a live Bittensor competition for continuously improving robotics models — const_reborn · 2026-07-29
- A microservice catalog test found Markdown, GraphRAG, and MCP graphs fail in different ways — JeremyCMorgan · 2026-07-29
- Google Research says a tiny brain sample exposed neurons linked by up to 50 synapses — GoogleAI · 2026-07-29
- Claude Tackles Quantum Info Open Problem, Says It Needs $1M in GPU Time — bronzeagepapi · 2026-07-29
- RLHF Book chapter breaks down PPO, GRPO and REINFORCE for LLM post-training — serrjoa · 2026-07-29
- What Anthropic’s alignment-faking paper shows — and what it does not — Passelume · 2026-07-29