Meta Expert: cross-layer expert sharing matches vanilla MoE with 70-80% experts
pmttyji · reddit · 2026-10-23
A new paper proposes Meta Expert, a cross-layer expert-sharing architecture for MoE models, targeting redundancy among topologically independent experts across layers.
- Empirical comparisons show Meta Expert models have lower parameter redundancy, lower cross-entropy loss and higher downstream accuracy.
- With activated expert count and granularity fixed, the model matches Vanilla MoE using only 70–80% of the experts; savings grow as models get deeper and larger.
- Theoretical analysis shows Meta Experts give hidden states more flexible residual pathways, improving information retrieval efficiency.
arXiv: 2610.27598
Related event: Meta Expert: cross-layer expert sharing matches MoE with 70-80% experts(3 posts)→
More from Research
- RouteWeaver: NTU Unifies Mode Selection and Execution in LLM Routing — NanyangTechnologicalUniversity · 2026-10-23
- HKU's TILDE Distills Embodied Foundation Models into Robot Policies — hkuhk · 2026-10-23
- RLSplat Cuts Feed-forward 3D Gaussian Splatting Memory 9.6x Linearly — Injae Kim · 2026-10-23
- Ant International's Heartbeat Thinking cuts voice-agent latency by 36.4% with speculative tool calls — ant-intl · 2026-10-23
- jessicata on why imperfect-recall games and Sleeping Beauty matter to decision theory — jessi_cata · 2026-10-23
- ReaDy Speculative Decoding Hits 3.16x Speedup on Qwen3.8-27B With a Single B200 — pmttyji · 2026-10-23