Moonshot’s Kimi K3 is a 2.8T open-weight MoE model with 1M-token context
alex_verem · x · 2026-07-29
Moonshot’s Kimi K3 technical report argues that the next leap in AI may come from using compute more efficiently, not just scaling it up.
What the model is
- Kimi K3 is a 2.8T-parameter open-weight MoE model with 104B activated parameters.
- It supports a 1-million-token context window.
- The architecture includes Kimi Delta Attention and Attention Residuals to improve information flow across sequence length and depth.
- Its Stable LatentMoE design activates only 16 of 896 experts per token, with the goal of routing compute where it matters instead of firing the whole model.
Why the team says it matters
The report claims roughly 2.5× better intelligence per unit of compute versus Kimi K2.
Reported results
Post-training reinforcement learning across general, coding, and reasoning tasks is said to improve compositional generalization and long-horizon execution.
At 2.8T scale, the system is presented as a combined effort in algorithm design, KDA-balanced expert-parallel training, memory management, million-token agentic RL, persistent rollout, sandbox states, and deployment innovations.
Benchmarks and positioning
- The report says Kimi K3 reaches frontier-level results in long-horizon coding, agentic tasks, knowledge, reasoning, and vision.
- It also claims the model trails only the strongest proprietary systems in their evaluated suite.
- Moonshot released the full weights to support future research and adoption.
More from Infra
- vLLM serves Beam at 1,769 tok/s on 4x B300 NVFP4, publishes Docker image and recipes — vllm_project · 2026-10-16
- SkyRL ships Day-0 RL support for Beam with +62% LoRA decode throughput in vLLM — vllm_project · 2026-10-16
- DFlash2 delivers up to 3.03x decode-only speedup for Beam across five benchmarks — vllm_project · 2026-10-16
- vLLM's Beam serving kernels cut GB200 forward time from 8.15 to 6.47 ms at TP8 batch 128 — vllm_project · 2026-10-16
- Inside Beam: 8-of-256 expert routing across 52 layers, sliding window plus NoPE attention — vllm_project · 2026-10-16
- vLLM ships Day-0 support for Reflection-Beam-1.0, a 501B-param MoE with 1M context — vllm_project · 2026-10-16