MoE-ViE: Mixture of Experts Vision Encoder for Efficient Image and Video Understanding
Bonan Zhang, Shiyu Dong, Quan Hung Tran, Katharina Gschwind, Shuqi Yang, Sijia Chen, Adel Ahmadyan, Seungwhan Moon, Lu Zhang, Ahmed Kirmani, Babak Damavandi, Anuj Kumar
ECCV 2026
cs.CV
2026-08-18
MoE-ViE swaps the CLIP vision tower for fine-grained experts plus shared experts and a custom Triton kernel; at every scale it beats dense, and the largest model matches a 1.9B dense encoder with 1.1B active parameters.
VLM vision towers gain accuracy with capacity, but dense scaling raises compute and latency linearly, and the bill compounds on high-resolution images and long videos. MoE already decouples capacity from compute in LLMs, yet every prior attempt at a CLIP-style MoE encoder (LIMoE, CLIP-MoE, CLIP-UP) fell short of SOTA dense models. The Meta team behind MoE-ViE asks whether the right topology lets a vision encoder be both cheaper and stronger.
Four decisions carry the paper.
Fine-grained experts. The conventional recipe replaces each dense MLP with N full-width experts for limited gains. Here each expert is 1/4 the dense MLP width, 32 experts per layer, so the same compute budget buys more, smaller experts. A width sweep shows consistent gains down to 1/4 width, saturating below. Visual features are heterogeneous (color, shape, spatial layout), and finer expert granularity matches that heterogeneity.
A shared expert. One always-on expert carries global context while routed experts apply conditional transformations; adding more shared experts stops helping beyond one.
Sigmoid routing with loss-free balancing. Sigmoid gating avoids expert competition. Balancing skips auxiliary losses entirely: a bias on the router is updated from observed token load, and instead of the standard constant sign step, the update scales with the z-score of each expert's load, so correction grows with imbalance and stops oscillating near balance. Both loss-free variants beat importance/load and entropy losses across the board.
A custom Triton kernel. Naive MoE implementations give back the theoretical savings to fragmented small GEMMs, CPU-GPU sync from dynamic routing, and HBM round trips. Grouped GEMM fuses all expert matrix multiplies into one GPU task; kernel fusion runs projections and activations in a single pass. At batch 16, latency drops from 318.76 ms to 82.59 ms, a 3.9x speedup.
The video stage fights forgetting. Pretraining runs on 3.5B image-text pairs; naive video finetuning then degrades image accuracy, and mixing image data back only partially recovers it while capping video gains. Their recipe: frame-level distillation (a frozen image-pretrained teacher constrains a randomly sampled frame via cosine loss) plus freezing the MoE experts and the text tower's MLP layers. A frame-based encoder's visual vocabulary needs little change for video; what changes is how frame features aggregate.
On zero-shot image benchmarks, MoE wins at every scale. Avg. classification rises from 55.8 to 63.2 at B/32 and 78.0 to 79.6 at L/16. The largest MoE-ViE-H/14 (3.5B total, 1.1B activated) against the 1.9B fully-activated PEcore G/14: ImageNet-A flips to a 0.6-point lead (93.2 vs 92.6), fine-grained average 83.9 vs 83.8, OCR average 80.2 vs 78.8, ImageNet 88.3 vs 88.6 slightly behind.
Aligned with an LLM the gap widens. With Llama 3.1 Instruct 8B, MoE-ViE-H averages 75.2 on image tasks where the best baseline (AIMv2 3B, 2.7B activated) gets 69.8, and 55.0 on video where every baseline sits below 47.6. At 1.1B activated parameters it passes InternViT2.5 at 5.5B activated.
Latency is the headline: at 576 input tokens MoE-ViE-H runs at roughly 76% of PEcore G's latency with comparable accuracy.
Experts do specialize. Layer-20 visualizations show expert 11 firing on a dog's ears, expert 18 on eyes and nose, expert 29 consistently on background. Routing entropy falls with depth (3.20 in layers 0-5 to 2.30 in layers 26-30 on the Flowers benchmark): deeper layers, sharper specialization.
This retires the idea that MoE only pays off in NLP: vision encoders can also get capacity for free via sparse activation. For VLM teams, a 1.1B-activated tower matching a 1.9B dense one opens a new tier for on-device and cost-sensitive deployments. The Triton kernel is a drop-in for other MoE topologies, and the fine-grained-plus-shared-expert recipe echoes DeepSeekMoE-style findings on the LLM side, so the design experience transfers across modalities.
For honesty's sake, an incremental improvement: absolute gains over the best dense models are mostly around one point. The paper's weight rests on the combination of consistency across scales, 42% fewer activated parameters, and 24% lower latency.
The paper has no dedicated limitations section; the appendix only covers implementation defects of naive MoE. Comparisons anchor on Meta's own PEcore family and SigLIP2, and 1.5B of the 3.5B pretraining pairs are proprietary, so independent verification rests on the released code and weights. After video finetuning, average video retrieval (49.5) trails PEcore G (49.9); video gains concentrate in classification. The B/L tiers activate 1/8 of experts versus 1/4 for the H tier, so cross-tier comparisons need care. Training cost is not reported, and MoE training and serving complexity exceeds dense.