Kimi K3 report says a 2.8T MoE used RL experts and multi-teacher distillation
cwolferesearch · x · 2026-07-29
A detailed breakdown of Kimi K3’s post-training recipe says the model is a 2.8T-parameter open-weight MoE with 104B activated parameters. The report describes a multi-stage pipeline: cold-start SFT with synthetic agentic trajectories and human-in-the-loop verification, per-domain RL across different effort levels, and then multi-teacher on-policy distillation to fuse nine expert teachers into one unified model.
The post argues that this recipe is becoming the standard pattern for frontier post-training: train narrow experts with RL, then consolidate them through distillation. It also notes that the report does not describe a separate RLHF or DPO stage, and claims Kimi K3 delivers around 2.5× overall scaling efficiency over Kimi K2, with strong results in coding, general reasoning, and long-horizon agentic tasks.
Related event: Kimi K3 shifts attention from scale to architecture(25 posts)→
More from Models
- Golden Gate Claude explainer resurfaces: interpretability premise still holds — HarperSCarroll · 2026-10-16
- Hugging Face livestreams release of a new open-weight model — huggingface · 2026-10-16
- Step 5 Preview hits #2 on OpenRouter weekly leaderboard with 19.5T tokens in one week — StepFun_ai · 2026-10-16
- Meta's open Muse Spark needs 493GB of memory — far beyond any consumer laptop — TejasKumar_ · 2026-10-16
- Apache 2.0 is the underrated part: Beam beats rival open drop on license — bigblueboo · 2026-10-16
- Apollo Research: GLM-5.3 spontaneously reasons about how its actions shape its future training data — Confident_Salt_8108 · 2026-10-16