Kimi K3 Architecture Revealed: KDA and LatentMoE Innovations
Ahmad_Al_Dahle · x · 2026-07-21
While awaiting the official tech report for Kimi K3, Ahmad Al-Dahle highlighted a few key architectural innovations:
- KDA: Swaps Gated DeltaNet's single scalar decay for learned per-dimension forgetting.
- AttnRes: Retrieves selectively across network depth.
- LatentMoE: Activates 16 out of 896 experts, balanced from router score quantiles.
Related event: Moonshot Releases 2.8T Open-Weights Model Kimi K3(14 posts)→
More from Research
- OpenAI and Apollo Research introduce Contrastive SDF to measure reward-seeking — OpenAI · 2026-07-22
- NVIDIA says to tune the harness before tuning the model with LangChain — NVIDIAAI · 2026-07-22
- The Thimble and the Waterfall: AI's Data Bottleneck and Feedback Loops — dyamins · 2026-07-22
- NVIDIA shows 22 SIGGRAPH papers and Omniverse tools for robot simulation — facontidavide · 2026-07-22
- Building a Knowledge Graph Without a Graph DB: 1000x Cheaper Than GraphRAG — TheRedfather · 2026-07-22
- Agentic RAG survey maps planner, retriever and refinement agents for complex retrieval — blaizedsouza · 2026-07-22