Mathematical Breakdown: Why Kimi K3 Abandons RoPE for Positional Encoding
nrehiew_ · x · 2026-08-08
This thread explains the underlying math of why the Kimi K3 model doesn't require RoPE (Rotary Positional Embedding) from a linear attention perspective.
- Essence of RoPE: Can be viewed as a product of accumulating transition matrices applied to queries and keys, equivalent to taking the matrix power of a static rotation matrix.
- Linear Attention Generalization: In linear attention, the transition matrix can change per position and become data-dependent. DeltaNet is commonly understood via read/write operations or online regression with fast weights.
- A New Perspective on GDN: By selecting the generalized Householder transform (acting as an erasing operation) as the transition matrix, we recover Generalized DeltaNet (GDN).
- KDA Replaces RoPE: As cited, DeltaNet inherently contains an orthogonal Householder matrix. Since powers of orthogonal matrices are proven to encode relative position, Kimi Delta Attention (KDA)—a DeltaNet variant—natively handles positional encoding, eliminating the need for RoPE.
More from Models
- GPT-5.6 Imitates User Personality, Enabling Memory Enhances Experience — Angaisb_ · 2026-08-09
- Grok's New Image Model Inherits GPT's Noise Artifacts via Synthetic Data Inbreeding — mark_k · 2026-08-08
- xAI Launches Grok Image 2.0: Achieving Precise Frame-by-Frame Image Generation — chaitu · 2026-08-08
- How to tell if an LLM's response is sincere or a corpus remnant — wavefnx · 2026-08-08
- User Complains LLMs Fail at Basic Data Pattern Recognition — Sarthak1411 · 2026-08-08
- Dev Rants About Claude's Overbearing Safety Refusals Hindering Side Projects — bankingyoung · 2026-08-08