Meta Expert: Cross-Layer Expert Sharing for Storage- and Retrieval-Efficient MoE
Ermo Hua, Xiangyu Hong, Baiting Wu, Che Jiang, Haozheng Hou, Yicheng Gu, Hanjing Wang, Qirui Zeng, Ming Zhang, Ning Ding, Qipeng Guo, Youbang Sun, Biqing Qi, Bowen Zhou
cs.AI
2026-10-20
Cross-layer Meta Experts keep per-token activation and expert width fixed, and match vanilla MoE loss with 70-80% of the experts. Deeper models save more stored parameters.
Sparse MoE already limits how much width each token computes. Along depth, mainstream models still give every layer its own expert set. Feed-forward layers store fairly static knowledge, and separate layers often store similar content, so Local Experts that never see each other still overlap in what they memorize.
Looped Transformers apply one block for several passes and can match much larger models with fewer parameters. Every extra pass adds serial depth and latency. Meta Experts keep a single forward pass. Parameters freed by sharing go into a larger candidate pool, and top-k routing holds the activated width and the FLOPs at the vanilla level. Shared experts are Meta Experts. Layer-private experts stay Local Experts.
A normal MoE layer has its own router and experts that only that layer can select. Pure Meta Experts collapse this to one shared router and one pool that every layer can read. The sweep starts from an all-local model at about 2B total parameters and 0.5B active. The meta share of the expert inventory is 0%, 25%, 50%, 75%, or 100%, and the meta share of per-layer activations is 25%, 50%, or 75%. Topology runs turn the front, the back, or both ends into local-only layers, and split the shared pool into 2, 4, or 8 sub-pools, read in chunks or interleaved. Total expert count and activations per layer stay locked so FLOPs match. Deeper runs reuse the compute-optimal vanilla hyperparameters, with no retune for sharing.
The kept recipe is MR75: 75% of experts sit in the shared pool, the two pools have separate routers, and each layer activates 4 meta plus 4 local. F2L2 makes the first two and last two layers local-only. Loop2 splits the shared pool into two interleaved sub-pools. Pure meta had lower pilot loss. The paper switches to 75% on the grounds that a fully shared pool might be unstable.
The shared pool changes how hidden states read and write. In one forward pass, a layer-wise MoE carries only content that earlier experts wrote into the residual stream. A deeper layer reaches shallow parameters only as a gradient, diluted by every layer in between, and those weights are private. Meta Experts add every layer's update into one parameter set. After training, each layer reads a superposition of those writes, on a path whose length does not grow with the distance between layers. The appendix is tighter than the introduction: inside a single forward pass, a shallow layer does not depend on the deep layer's activations from that same step. It reads content the deep layer wrote into the pool during training.
The scaling ladder is a Qwen3.5-style text-only MoE at depths 12, 16, 20, and 24, with width grown alongside depth so each rung is near compute-optimal. Each layer has 128 routed experts plus one gated shared expert and top-8 routing, mixing linear attention with full attention. The 16-layer vanilla backbone has about 1.96B parameters and activates about 449M per token. Training is from scratch with Muon on H200s. The four rungs use about 5.31B, 11.06B, 20.47B, and 34.92B tokens, and about 7.1×10^18 to 2.4×10^20 FLOPs. Loss is the training loss averaged over the last ten steps, plotted as a ratio to vanilla.
A higher meta share lowers both cross-entropy and expert redundancy. Giving more of the activation budget to local experts lowers redundancy and worsens loss. Long local-only stretches, and finely split pools, usually hurt. Keeping only the first two and last two layers local, with the pool split in two, helps a little. Stacking MR75, F2L2, and Loop2 improves loss and redundancy at every compute rung, and the gap widens as scale grows. At a matched expert budget the loss ratio stays under 1, on a figure whose axis bottoms at 0.990.
Meta width is then cut down from 96 per layer, and the freed slots are not filled back in with local experts. Top-8 and expert width stay fixed, so activated FLOPs stay fixed while stored experts fall. The text puts the saving at about 20-30% of the experts, or 15-25% of parameters. Figure 1 draws the same comparison as 70-80% of the experts, or 75-90% of the parameters, with loss matching vanilla. Deeper models save more.
The downstream run is separate: 6B total parameters, 1.6B active, 200B tokens. Five variants share one expert inventory and one per-layer activation budget.
| Method | MMLU | C-Eval | CMMLU | TriviaQA | BBH | OpenBookQA |
| Vanilla | 40.87 | 43.54 | 42.03 | 19.24 | 29.72 | 35.00 |
| MR75 | 41.20 | 44.43 | 41.61 | 19.64 | 29.12 | 33.80 |
| MR75+Loop2 | 41.75 | 44.87 | 43.44 | 19.32 | 30.30 | 36.40 |
| MR75+F2L2 | 39.28 | 41.60 | 41.98 | 20.14 | 30.61 | 33.20 |
| All three | 40.52 | 43.02 | 42.49 | 20.42 | 29.84 | 35.40 |
MMLU, C-Eval, and CMMLU are 5-shot accuracy. OpenBookQA is normalized accuracy. BBH and TriviaQA are generation exact match. The combination leads on the loss curves and does not lead the table. MR75+Loop2 wins the knowledge suites: MMLU 41.75 versus 40.87, C-Eval 44.87 versus 43.54, CMMLU 43.44 versus 42.03. TriviaQA prefers the combination, 20.42 versus 19.24. F2L2 alone drops MMLU to 39.28 and C-Eval to 41.60.
The saving is storage. Each token still activates eight experts, so training and inference compute match vanilla, while expert weights can shrink by about a fifth to a third. Expert-parallel jobs move less weight. The forward pass is still one pass, and attention can stay layer-private, so the latency stays that of a standard MoE step. Leaving the ends local matches the finding that the shallowest and deepest layers are not redundant with each other. At a matched budget the loss move is a small notch, most downstream deltas are about one point, and the gain grows with depth. This is an incremental architecture change.
Earlier cross-layer MoE sharing usually puts every layer on a single pool, which is hard to scale in communication and memory. Here, which layers share, and how many sub-pools exist, are topology choices that can be swept.
The ladder stops at 24 layers and about 2.4×10^20 FLOPs. The 6B-A1.6B downstream run sees 200B tokens. Larger models are left open.
Pure meta had the best pilot loss, and it was dropped for a stability risk the paper does not show. Mild hierarchy raises redundancy without raising loss, which the paper leaves as an open question. The introduction's backward residual, described as acting inside one forward pass, is stronger than the appendix, which says the causal direction of that pass is unchanged. Every variant inherits the vanilla learning rate, batch size, and token budget, so the shared topology never receives its own compute-optimal setup. Redundancy is a Gini coefficient on the spectrum of a linear-CKA kernel over expert responses. The settings that lower it are not the settings that win downstream.