Kimi K3 Architecture: Dropping RoPE for KDA and LatentMoE
khademinori · x · 2026-07-30
The thread explores the key architectural trade-offs in the Kimi K3 model and how they translate to strong performance.
- Core Architecture: K3 drops RoPE entirely, even though its original author is on the team. Instead, it uses Kimi Delta Attention (KDA), which acts as a recurrent, content-addressable matrix-state RNN, making tokens order-aware before global MLA.
- New Components: The main addition compared to Kimi Linear is the LatentMoE, which compresses large linear layers via down-projection, similar to Nemotron 3 Ultra.
- Scaling Success: The architecture of Kimi Linear was successfully scaled from 48B to a 2.8T production-scale frontier model, proving its stability under extreme scaling.
Related event: Kimi K3 Open-Source Model Architecture: Dropping RoPE for KDA and LatentMoE(5 posts)→
More from Models
- Golden Gate Claude explainer resurfaces: interpretability premise still holds — HarperSCarroll · 2026-10-16
- Hugging Face livestreams release of a new open-weight model — huggingface · 2026-10-16
- Step 5 Preview hits #2 on OpenRouter weekly leaderboard with 19.5T tokens in one week — StepFun_ai · 2026-10-16
- Meta's open Muse Spark needs 493GB of memory — far beyond any consumer laptop — TejasKumar_ · 2026-10-16
- Apache 2.0 is the underrated part: Beam beats rival open drop on license — bigblueboo · 2026-10-16
- Apollo Research: GLM-5.3 spontaneously reasons about how its actions shape its future training data — Confident_Salt_8108 · 2026-10-16