Kimi K3 scales Kimi Linear to 2.8T parameters and drops RoPE for NoPE
rasbt · x · 2026-07-28
Sebastian Raschka breaks down the Kimi K3 architecture from yesterday’s open-weight release and highlights a few notable changes:
- Scale-up of Kimi Linear: K3 is essentially a much larger production version of Kimi Linear, growing from 48B to 2.8T parameters.
- LatentMoE added: The main new component is LatentMoE, similar to the one used in Nemotron 3 Ultra, designed to compress large linear layers.
- Efficiency-first design: Like Nemotron 3 and DeepSeek V4, K3 leans heavily toward better inference efficiency, replacing standard components with optimized variants such as MoE → LatentMoE, regular attention → multi-head latent attention, and Kimi Delta Attention.
- Residual-path tweak: The architecture also introduces attention residuals, which improve validation loss and downstream performance at the cost of about 4% extra training compute and 2% extra inference cost.
- NoPE everywhere: K3 removes RoPE entirely and uses NoPE across the stack, which Raschka notes is the first frontier-level model he has seen with this setup.
- Native multimodal support: The release also adds built-in multimodal capability.
He closes by calling the release a very strong architecture-level update.
Related event: Kimi K3 shifts attention from scale to architecture(25 posts)→
More from Models
- Golden Gate Claude explainer resurfaces: interpretability premise still holds — HarperSCarroll · 2026-10-16
- Hugging Face livestreams release of a new open-weight model — huggingface · 2026-10-16
- Step 5 Preview hits #2 on OpenRouter weekly leaderboard with 19.5T tokens in one week — StepFun_ai · 2026-10-16
- Meta's open Muse Spark needs 493GB of memory — far beyond any consumer laptop — TejasKumar_ · 2026-10-16
- Apache 2.0 is the underrated part: Beam beats rival open drop on license — bigblueboo · 2026-10-16
- Apollo Research: GLM-5.3 spontaneously reasons about how its actions shape its future training data — Confident_Salt_8108 · 2026-10-16