Kimi K3 drops RoPE for NoPE, but recurrent KDA still encodes order
cedric_chee · x · 2026-07-29
Kimi K3 reportedly removes all RoPE layers and uses NoPE, but it is not truly position-blind.
- KDA is recurrent: the model keeps a compact fast-weight memory and updates it with a delta-rule style mechanism, more like a content-addressable matrix-state RNN than standard softmax attention.
- Order is implicit: gating, decay, and causal masking still encode sequence direction and recency.
- MLA handles retrieval: global content retrieval comes from MLA without explicit positional encoding.
- Stacking matters: the post notes that K3 places three KDA layers before each gated MLA layer, including before the first one, so tokens already carry order-aware representations before retrieval layers kick in.
Related event: Kimi K3 Architecture Drops RoPE for NoPE(3 posts)→
More from Models
- theo builds his own visualizer for today's agent models, showing how cheap Luna really is — ivan_bezdomny · 2026-09-23
- Why ChatGPT Still Wins: One User's Split Between Muse, Claude and Codex — mobileraj · 2026-09-23
- Muse reportedly offers 4B tokens/week for ~$100/month, sparking industry price-disruption talk — NewYak4281 · 2026-09-23
- GPT-6 Sol and Luna appear in OpenAI docs, alongside guidance on reasoning effort — cedric_chee · 2026-09-23
- GPT-6 tested on LIBERO robot task: turns on stove, fails to grasp moka pot — YuXiang_IRVL · 2026-09-23
- Ternary Bonsai 2 27B: 5.9GB weights retain ~95% of full-precision reasoning — cephaloform · 2026-09-23