A 671B model only activates ~37B per token: how MoE works and what it costs

techNmak · x · 2026-09-12

A long-form explainer on why a 671B-parameter model doesn't necessarily use 671B parameters for your next token:

The trick: replace each layer's single feed-forward network with many "expert" subnetworks plus a tiny router that picks which few handle the current token. Specialization is learned, not hand-designed.
Example: DeepSeek-V3 has 671B total parameters but activates only 37B per token — sparse MoE scales parameter capacity without proportionally scaling compute.
The overlooked catch: inactive doesn't mean nonexistent. Weights still must be stored across accelerators; routing tokens to the right experts adds communication overhead, and balancing expert utilization is an engineering problem of its own.
Bottom line: a 671B MoE isn't a secretly small model — it's a 671B model that learned which parts of itself to wake up for each token.

More from Models