Kimi K3 Open-Source Model Architecture: Dropping RoPE for KDA and LatentMoE

Kimi K3 is the largest open-source model with 2.8 trillion total parameters and only 104 billion activated per inference. The model makes a significant architectural trade-off by completely abandoning RoPE (rotary position encoding) in favor of Kimi Delta Attention (KDA) and LatentMoE to address the slowdown caused by memory accumulation in long-context processing. The community has not only dissected the technical details but also traced the research lineage, suggesting that K3 is not a single breakthrough but a convergence of multiple technical lines.

Confirmed

• Model scale: Kimi K3 has 2.8T total parameters and activates 104B per inference.
• Core architecture: The model completely abandons RoPE, despite its original author's involvement. It replaces it with Kimi Delta Attention (KDA) and LatentMoE.
• Attention mechanism: KDA is essentially a form of linear or hybrid attention, incorporating attention residual designs first seen in Kimi Linear. KDA aims to solve long-context memory accumulation slowdown via mechanisms like "active forgetting."

Why it matters

• @East-Muffin-6472's reading roadmap points out that Kimi K3 is not a single breakthrough but a convergence of multiple technical lines. Its research stack traces back to early papers like "Linear Transformers Are Secretly Fast Weight Pro..." Understanding these prerequisite studies helps see through K3's technical essence.

2026-07-29 ~ 2026-07-31 · 5 related posts

• [source] A reading order for Kimi K3 traces the papers behind Moonshot’s frontier model — East-Muffin-6472 · 2026-07-29
• Kimi K3 Architecture: Dropping RoPE for KDA and LatentMoE — khademinori · 2026-07-30
• [source] Kimi K3 Architecture Explained: Building a 2.8T Parameter Open Model via 'Active Forgetting' — AccBalanced · 2026-07-31
• [source] Analyzing Kimi K3 Attention: Delta Attention and Residuals — AndLukyane · 2026-07-31

1 near-duplicate retellings: East-Muffin-6472