Deep Dive into Kimi K3's Sparse MoE Details
techNmak · x · 2026-07-19
This in-depth post explains the sparse MoE design of Kimi K3: it boasts 2.8 trillion parameters, but during a single sparse expert computation, it only activates 16 out of 896 experts, leaving the vast majority inactive for that pass.
The author points out that the challenge isn't just the number of experts, but rather routing and load balancing: if a few experts are over-assigned, throughput will tank. Moonshot's solution is Quantile Balancing, which directly uses the quantiles of routing scores for allocation, avoiding repetitive heuristic tweaks and an overly sensitive balancing hyperparameter. Additionally, they implemented fully balanced expert-parallel training using static shapes without requiring host synchronization on the critical path, aiming to prevent expert imbalance from degrading distributed training throughput.
More from coding & agent
- Gergely Orosz: Shipping 10x PRs With AI Agents, Sites Fill With Small Regressions — ducha_aiki · 2026-09-11
- Same Echo Maze prompt, three frontier models: all passed visually but shipped the same hidden bug — eyishazyer · 2026-09-11
- Astra storyboards plus Minimax H3 per-shot generation boost video success rates — Hailuo_AI · 2026-09-11
- Codex tip: use Sol with Astra and Luna sub-agents to save usage — pvncher · 2026-09-11
- agents-best-practices: a provider-neutral Agent Skill for designing and auditing agentic harnesses — tom_doerr · 2026-09-11
- Cognition's SWE-2 uses a KKT duality argument in RL to shift the effort Pareto curve — YouJiacheng · 2026-09-11