MoE handbook explains routing, load balancing, z-loss and expert parallelism in depth
Franc0Fernand0 · x · 2026-09-15
A developer published a handbook explaining how Mixture of Experts really works, starting from what happens when a Transformer FFN becomes a routed bank of experts with only a small subset activated per token.
- Routing mechanics: router logits, Top-k selection, expert weighting
- Load balancing: models are trained to pick the best expert, so an auxiliary loss penalizes over-concentration—the part the author finds most confusing
- Capacity & dropping: expert capacity, token dropping, and dropless execution
- Training stability: router z-loss
- Parameters: total vs. active parameters
- Parallelism: expert parallelism
A solid systems-level primer for understanding the sparse architectures behind models like DeepSeek and Llama.
Related event: Developer's MoE Handbook Goes Viral: From Routing to Expert Parallelism(3 posts)→
More from Research
- Sakana's PC-ALM trains 1000-layer nets without backprop; LeCun calls it target prop reborn — ylecun · 2026-09-15
- Five exercises find four defects in the author's own LLM-judge eval system — alexpran · 2026-09-15
- How a Major Children's Hospital Uses Open Source NVIDIA AI for Cardiac Care — nordicinst · 2026-09-15
- CMU-led preprint frames strategic motor learning as hypothesis testing — AnnaCiaunica · 2026-09-15
- Reasoning-Filtered Dataset Fable-5.1-Max-Filtered-5000x Trends on Hugging Face — MoreThought · 2026-09-15
- AISTATS 2027 heads to Montreal, paper submissions open September — qberthet · 2026-09-15