MoRE: Sharing expert pools across adjacent layers cuts perplexity at matched params
burny_tech · x · 2026-10-30
A Cornell team (incl. Kilian Weinberger) proposes MoRE: Mixture of Reused Experts: adjacent Transformer layers share a larger expert pool while each layer keeps its own router, with lightweight learnable depth embeddings conditioning layer inputs so shared experts can distinguish layers.
Motivation: standard MoE's expert params grow linearly with expert count (memory cost), while recurrent weight-sharing Transformers lack capacity. Across three scales (114M–1.15B), MoRE consistently beats standard MoE and SOTA weight-sharing architectures at matched compute/params: at 1.15B, perplexity drops from 11.88 to 11.75 and avg downstream accuracy rises from 47.05% to 48.12%, requiring only minimal changes to existing MoE implementations.
More from Research
- Investor's Last Exam Benchmark: Top AI Models Ace Research but Fail at Investment Judgment — Huangfei Jiang · 2026-10-30
- SpecSlot Boosts Pixel-Space Image Generation: +2.84 FID on ImageNet-512 at Lower Cost — Jiongfan Yang · 2026-10-30
- Apple Trains UI Navigation Agents with Online RL Directly on Live iOS Apps — apple · 2026-10-30
- KAIST's L3G-World Generates Infinite Interactive Worlds in Real Time with Latent 3D Gaussian Memory — kaist-ai · 2026-10-30
- PhysInsert from Beihang and Meitu Achieves Physically Plausible Reference-Based Object Insertion — Yue Gong · 2026-10-30
- AB-Attention Speeds Up Video Diffusion Transformers with Training-Free Accurate Block-Sparse Attention — Bin Xu · 2026-10-30