Stanford CS336 lecture notes unpack MoE scaling: 64 experts stored, still 2P active per token
stanfordnlp · x · 2026-10-16
Chao Ma published detailed notes on Stanford CS336 (Spring 2026) Lecture 4, taught by Tatsunori Hashimoto, covering attention alternatives and mixture-of-experts.
Key points:
- Two separate bills in MoE: growing an expert bank from 8 to 64 equal-size experts (top-k fixed at 2) raises stored parameters from 8P to 64P, yet each token still activates only 2P expert parameters. More capacity is not free: routing, shared layers, and communication cost real compute, and a batch may collectively activate every expert.
- Contrast with recurrent attention states: recurrent states compress growing history into a fixed-size buffer, reducing memory cost over time — MoE changes per-token feedforward compute, while recurrent states change how information is carried across time.
- The lecture unifies sparse attention, recurrent states, and conditional computation under one question: which work must happen per token, and which information must remain available.
The notes include the O(n²dk+n²dv) per-head attention cost derivation; numerical examples are constructed, not benchmark results. Official recording and slides are public.
More from Models
- Sentience claims per-user models: train, continually update and serve one model per person — soleio · 2026-10-17
- Four Gemini 4 Argon agents beat Minecraft in 27 hours, writing 2,783 programs — GoogleDeepMind · 2026-10-17
- Meta open-sources Muse Spark Open to mixed reception: 'trash' license, nerfed benchmark scores — gabriberton · 2026-10-17
- If new diffusion models are open-weight only, what's left for commercial use? — blackpanda8484 · 2026-10-17
- Reflection Beam OSS architecture: sliding window attention, gated attention, QK Norm — A_K_Nain · 2026-10-17
- LLMs play a 2D platformer as a decision-making benchmark; 7.9B model only one to clear all 14 levels — Felladrin · 2026-10-17