Mathematical Breakdown: Why Kimi K3 Abandons RoPE for Positional Encoding
nrehiew_ · x · 2026-08-08
This thread explains the underlying math of why the Kimi K3 model doesn't require RoPE (Rotary Positional Embedding) from a linear attention perspective.
- Essence of RoPE: Can be viewed as a product of accumulating transition matrices applied to queries and keys, equivalent to taking the matrix power of a static rotation matrix.
- Linear Attention Generalization: In linear attention, the transition matrix can change per position and become data-dependent. DeltaNet is commonly understood via read/write operations or online regression with fast weights.
- A New Perspective on GDN: By selecting the generalized Householder transform (acting as an erasing operation) as the transition matrix, we recover Generalized DeltaNet (GDN).
- KDA Replaces RoPE: As cited, DeltaNet inherently contains an orthogonal Householder matrix. Since powers of orthogonal matrices are proven to encode relative position, Kimi Delta Attention (KDA)—a DeltaNet variant—natively handles positional encoding, eliminating the need for RoPE.
Related event: Mathematical Analysis of Kimi K3: Why RoPE Is Dropped(2 posts)→
More from Models
- GLM 5.3 Flash scores 39 on Artificial Analysis index, beating larger Mistral and Muse models — airesearch12 · 2026-10-16
- Open-source ASR built on a Qwen3-4B decoder posts 3.54 average WER across 5 languages — rohanpaul_ai · 2026-10-16
- Google's "Paid Pro" paywall in Antigravity angers Pixel buyers and bundle subscribers — Chance-Okra-5920 · 2026-10-16
- GLM 5.3 Flash is now free in OpenCode coding tool — DanielLockyer · 2026-10-16
- GPT-6 Sol runs a potion shop flawlessly but its reasoning mistakes stock for its own stats — liminal_bardo · 2026-10-16
- Codex convert says Claude app is 2-3x better optimized with a far smoother UI — ItchyResearcher5671 · 2026-10-16