Kimi K3 drops RoPE for NoPE, but recurrent KDA still encodes order
cedric_chee · x · 2026-07-29
Kimi K3 reportedly removes all RoPE layers and uses NoPE, but it is not truly position-blind.
- KDA is recurrent: the model keeps a compact fast-weight memory and updates it with a delta-rule style mechanism, more like a content-addressable matrix-state RNN than standard softmax attention.
- Order is implicit: gating, decay, and causal masking still encode sequence direction and recency.
- MLA handles retrieval: global content retrieval comes from MLA without explicit positional encoding.
- Stacking matters: the post notes that K3 places three KDA layers before each gated MLA layer, including before the first one, so tokens already carry order-aware representations before retrieval layers kick in.
Related event: Kimi K3 Architecture Drops RoPE for NoPE(3 posts)→
More from Models
- DeepSeek v4 Flash thinking mode silently ignores sampling controls in API runs — OkDimension2228 · 2026-07-29
- Polymarket launches a market on when Anthropic’s next Claude Opus will ship — Polymarket · 2026-07-29
- Rumor says DeepSeek V4 is in internal testing and could hit GA in early August — teortaxesTex · 2026-07-29
- Debate heats up over whether Chinese models can catch the U.S. frontier before 2030 — teortaxesTex · 2026-07-29
- Microsoft’s Mage-Flow models vanish from Hugging Face, but community mirrors remain — pmttyji · 2026-07-29
- Liquid AI plans a talk on its upcoming edge agentic model and harness design — maximelabonne · 2026-07-29