Kimi K3 Architecture and 2.8T Scale
scaling01 · x · 2026-07-16
This share notes that Kimi K3 incorporates two architectural designs: Kimi Delta Attention (KDA) and Attention Residuals (AttnRes).
Additionally, cited sources confirm that Moonshot AI has verified the Kimi scale reaches 2.8T. The focus here is strictly on model architecture and scale, rather than application layers.
Related event: Kimi K3 Debuts Strong, Narrowing the Open-Weight Gap(184 posts)→
More from Models
- Cohere's Invent-A-Dataset paper: zero-seed synthetic data beats frontier APIs on quality and diversity — idanbeck · 2026-10-27
- ARC Prize redesigns leaderboard: highest reasoning effort by default, CSV export — mhmazur · 2026-10-27
- Cohere: open-weight Qwen3.8-Flash-Next tried uploading probe file to Beijing cloud bucket in sandbox — wstone_bd · 2026-10-27
- Model refuses wrong psychiatric advice in deliberately misleading test — FellMentKE · 2026-10-27
- Clinical System One v0.2: top clinical AI drops from 97% to 65% on harder cases — MaziyarPanahi · 2026-10-27
- ChatGPT keeps telling people 404 Media is tech support for big corporations — 404 Media · 2026-10-26