Deep Dive into Kimi K3's Sparse MoE Details
techNmak · x · 2026-07-19
This in-depth post explains the sparse MoE design of Kimi K3: it boasts 2.8 trillion parameters, but during a single sparse expert computation, it only activates 16 out of 896 experts, leaving the vast majority inactive for that pass.
The author points out that the challenge isn't just the number of experts, but rather routing and load balancing: if a few experts are over-assigned, throughput will tank. Moonshot's solution is Quantile Balancing, which directly uses the quantiles of routing scores for allocation, avoiding repetitive heuristic tweaks and an overly sensitive balancing hyperparameter. Additionally, they implemented fully balanced expert-parallel training using static shapes without requiring host synchronization on the critical path, aiming to prevent expert imbalance from degrading distributed training throughput.
More from coding & agent
- An MCP server signs every AI agent tool call into a verifiable Merkle chain — Funky_Chicken_22 · 2026-07-22
- Claude Code skill uses 10 Markdown rules to make outputs ADHD-friendly — alex_verem · 2026-07-22
- A Firecracker-based platform says it can host 6,000 AI agents on one 256 GB server — maritime_sh · 2026-07-22
- A better path to agent autonomy is running waves, finding friction, and iterating — JnBrymn · 2026-07-22
- AI agent designers map the visual and tonal cues behind companionship products — Unlikely-Platform-47 · 2026-07-22
- Coding agents are heading toward an AI-writes, AI-reviews, human-approves workflow — aftahi_ai · 2026-07-22