The Myth of Expert Specialization in MoEs: Why Routing Reflects Geometry, Not Necessarily Domain Expertise
Xi Wang, Soufiane Hayou, Eric Nalisnick
cs.AI
2026-04-11
MoE routers are linear, so expert use tracks hidden-state geometry. On five models, cross-model overlap for the same contest problem is only ~60%, matching a different problem.
MoE layers replace each dense feed-forward block with a bank of experts and run only the top-k for every token. Parameter count goes up while compute per token stays closer to a smaller model. Model reports keep showing code, math, and web text lighting up different experts, and that figure is easy to read as domain expertise.
The router is a matrix P. Expert logits are P times the token hidden state h. Nearby hidden states must produce nearby logits. Distant ones may or may not. Specialization tracks where inputs sit in representation space. The router only projects that geometry onto expert scores. The paper turns the relation into a bound and checks it on five released MoEs.
Proposition 1 bounds logit distance between two tokens. After an SVD of the hidden states, the top r energy directions form a principal subspace. Logit distance is at most the router's gain on that subspace times the hidden-state distance inside it, plus a residual from the leftover directions. When variance sits in the leading directions, the residual is negligible, so hidden states that are close along those directions must share experts. Distant hidden states have no such guarantee. On the middle layer of five models, 16 OpenWebText sequences each, the scatter is a triangle: close hidden states sit with close logits, and far hidden states go either way. Outliers are distances to the first token, an attention sink with an abnormally large activation.
Proposition 2 covers the load-balancing loss, the auxiliary term that pushes expert frequencies toward uniform. A shared direction μ in the hidden states is mapped near a constant by a near-optimal router. Softmax and top-k ignore a constant shift, so that direction drops out of the choice. In a small batch the shared direction changes every step, and the router never locks onto stable separating directions. That accounts for the collapse Qiu et al. saw when balancing runs on tiny micro-batches: different topics fall into almost the same experts. This paper does not retrain a model.
Depth raises correlation inside a sequence. On gpt-oss-20b, trained with AdamW and an auxiliary loss, hidden-state cosine similarity keeps rising after about layer 11, while router-output similarity stops following and energy on the first singular direction is pushed down. Trinity-Mini-Base, trained with Muon and no auxiliary loss, shows neither rising hidden-state correlation nor this suppression. Moonlight 16B, also auxiliary-loss-free, does not show the suppression.
Context rewrites routing in deeper layers. On gpt-oss-20b the word apple shares more experts between an everyday sentence and a code line in layer 1 than later. Context-free specialization reported for OpenMoE looks like under-training. At sequence scale, 100 OpenWebText documents and 100 from Nemotron-CC-Math show the same pattern on all five models at the middle layer: more similar sequences use more similar expert frequencies, cross-dataset pairs have lower hidden-state similarity, and dissimilar sequences can still share experts.
On 30 HMMT February 2025 problems, the comparison uses solutions from 60 reasoning models, with routing measured on Qwen3-30B-A3B, ERNIE-4.5-21B-A3B, and gpt-oss-20b, four seeds each. Overlap is Jaccard similarity at Top-p=0.8, agreement among experts that cover 80% of tokens. Same model, same question, different seeds is the high end. Same model on a different question, and different models on the same question, both land near 60% in every layer. Other cutoffs keep the shape.
Three paired prompts on Ling-mini have nearly the same expert frequencies during prefill, the forward pass over the prompt before generation: different concepts in one style, one concept in plain versus mathematical language, and AIME problems in Chinese versus French. After generation starts, some pairs stay aligned and some drop. Prefill routing does not predict which.
Deep prefill on gpt-oss-20b can assign the same experts to unrelated inputs. A 411-token HLE question and a 2,500-token history of Wimbledon light up nearly the same experts in later layers. Keeping the 12 most-used experts out of 32 and dropping them from the last two layers raises negative log-likelihood by less than 10% on prompt-only text across code, generic text, economics, and HLE questions. Once on-policy or off-policy completions are included, the dropped experts affect conditional likelihood again. Matching frequency curves split as the rollout grows, earlier in deeper layers. The non-reasoning model ERNIE-4.5-21B-A3B-PT does not collapse this way. Removing the chat template reduces the gpt-oss collapse. Duplicating the input from 1× to 12×, template excluded, pushes cross-domain expert use toward the in-domain baseline.
Shuffling or reversing 16 OpenWebText sequences lowers maximum softmax probability on ERNIE-4.5-21B, gpt-oss-20b, and Qwen3-30B.
| Comparison | Metric | Result |
| Same question across models vs a new question | Jaccard at Top-p=0.8 | Both about 60%, below same question and model |
| Keep 12 of 32 experts, drop last 2 layers | Prompt-only NLL | Increase under 10% |
| Same pruning, plus a rollout | Conditional NLL | Clearly worse; no single percentage |
| Shuffled or reversed vs original | Max softmax | Lower on all three models |
At corpus scale the geometry is enough to use. Different domains sit in different parts of hidden-state space, so they hit different experts. The domain plots in the DeepSeek-V3 and OLMoE reports fit that picture. They also remain a redundancy check: if every input enters the same experts, the extra parameters are copies.
One problem or one rollout is a different regime. Audits that look only at the prompt will read deep prefill collapse as shared expertise. Reasoning models make that mistake easier, because their opening tokens are largely a template. Hidden-state geometry is still unsolved in dense models. Expert specialization is at least that hard, with an extra many-to-one: different hidden states can share experts, and similar ones can split once generation starts. Without an extra training objective, logit space will not line up with human categories.
Proposition 2 assumes small logits and a linearized softmax, and the balancing loss is simplified. In a real objective it is only one term, so shared directions are only partly suppressed. Released models do not show a router blind to the data's leading direction. The paper says weak models of that kind are usually not released, so the sample favors routers that already work. The small-batch collapse is Qiu et al.'s experiment, not a training curve in this paper. The out-of-distribution test is only shuffling and reversal, with no detection score.
The 60% figure covers three reasoning MoEs, 30 HMMT problems, and Top-p=0.8. The figure is split by those three MoEs, while the text also mentions 60 reasoning models. Expert indices are not a shared vocabulary across architectures, and the paper never says whether Jaccard runs external solutions through one router or intersects expert ids from different models.
The abstract calls hidden-state similarity necessary and sufficient for similar expert use. Figure 1 supports the sufficient half, and only when the residual is small. Distant hidden states often still produce similar logits. Later sections state that many-to-one. The abstract is stronger than the plot.