Switch Transformer by hand: a 13-step walkthrough of how sparse MoE works
ProfTomYeh · x · 2026-09-10
ProfTomYeh shares a hand-calculation walkthrough of the Switch Transformer (Fedus, Zoph & Shazeer, 2022), the paper that made sparse Mixture of Experts practical.
- 13 steps run five features through attention pooling, then a switch matrix produces gate values for experts A/B/C
- Each feature routes only to its top-scoring expert (sparsity), experts have fixed capacity 2, and overflow features pass through unchanged
- Attention stays intact; the dense FFN is replaced by a sparse expert set — most parameters live in the experts but only a small slice activates per token
- That's the mechanism letting GPT-4, Claude, DeepSeek-V3 and Kimi store huge parameter counts while keeping inference cheap
More from Models
- NeoHorse-1-4B, a Qwen3.5-based agentic model, trends on Hugging Face — TokenRhythm · 2026-09-10
- Chinese model 3D face-off: DeepSeek V4.1 Flash crushes Kimi K3 and GLM-5.3 — teortaxesTex · 2026-09-10
- DeepSeek 4.1 Flash spotted running the Boeing bench, results pending — victormustar · 2026-09-10
- Sentry CEO: switch off the priciest reasoning-tier models — you won't notice a performance difference — zeeg · 2026-09-10
- Latest Models Are Now Surprisingly Good at Driving ffmpeg — Flomerboy · 2026-09-10
- Astra and Fable 5.1 benchmarks barely overlap, making leaderboard comparisons misleading — recro69 · 2026-09-10