Kimi K3 scales Kimi Linear to 2.8T parameters and drops RoPE for NoPE
rasbt · x · 2026-07-28
Sebastian Raschka breaks down the Kimi K3 architecture from yesterday’s open-weight release and highlights a few notable changes:
- Scale-up of Kimi Linear: K3 is essentially a much larger production version of Kimi Linear, growing from 48B to 2.8T parameters.
- LatentMoE added: The main new component is LatentMoE, similar to the one used in Nemotron 3 Ultra, designed to compress large linear layers.
- Efficiency-first design: Like Nemotron 3 and DeepSeek V4, K3 leans heavily toward better inference efficiency, replacing standard components with optimized variants such as MoE → LatentMoE, regular attention → multi-head latent attention, and Kimi Delta Attention.
- Residual-path tweak: The architecture also introduces attention residuals, which improve validation loss and downstream performance at the cost of about 4% extra training compute and 2% extra inference cost.
- NoPE everywhere: K3 removes RoPE entirely and uses NoPE across the stack, which Raschka notes is the first frontier-level model he has seen with this setup.
- Native multimodal support: The release also adds built-in multimodal capability.
He closes by calling the release a very strong architecture-level update.
Related event: Kimi K3 Architecture Recap: Seven Years of Evolution Beyond Scaling(6 posts)→
More from Models
- Kimi K3 reportedly runs on 80 RTX 5090s with 2.56 TB of VRAM — sandyyevans · 2026-07-28
- Users report Opus 5 starts sounding like Trump in base model mode — Sauers_ · 2026-07-28
- Kimi is described with a linear-attention variant and attention residuals — burny_tech · 2026-07-28
- Reddit users say OpenAI’s Sol may have degraded after release — unhappinessNvrCame · 2026-07-28
- GPT-5.6 sol is described as a full coding package for medium-to-large codebases — haider1 · 2026-07-28
- Moonshot AI reportedly seeks advanced Nvidia chips for Kimi K4 training — Polymarket · 2026-07-28