Deep Dive into Kimi K3's Sparse MoE Details
techNmak · x · 2026-07-19
This in-depth post explains the sparse MoE design of Kimi K3: it boasts 2.8 trillion parameters, but during a single sparse expert computation, it only activates 16 out of 896 experts, leaving the vast majority inactive for that pass.
The author points out that the challenge isn't just the number of experts, but rather routing and load balancing: if a few experts are over-assigned, throughput will tank. Moonshot's solution is Quantile Balancing, which directly uses the quantiles of routing scores for allocation, avoiding repetitive heuristic tweaks and an overly sensitive balancing hyperparameter. Additionally, they implemented fully balanced expert-parallel training using static shapes without requiring host synchronization on the critical path, aiming to prevent expert imbalance from degrading distributed training throughput.
More from coding & agent
- 'Let an agent run wild on vague goals and it makes your code worse 9 out of 10 times' — eptwts · 2026-10-27
- LangChain to demo the in-house design agent its team uses daily, this Wednesday — LangChain · 2026-10-27
- Biotech Agent Series Day 8: A Notebook Review Agent to Auto-Audit Lab Records — nlarusstone · 2026-10-27
- Photon Launches Agent-to-Agent Messaging in iMessage for Real AI Collaboration — alvelda · 2026-10-27
- Fake-paper meme shows an agent workflow for reverse engineering trading systems into a one-person quant fund — PtrPomorski · 2026-10-27
- Codex clones Linear's core from a one-page spec in 3 minutes 28 seconds — aakashgupta · 2026-10-27