Meta's MoEMB: MoE scaling beats 4x-larger embedders with just 3B active params
_reachsumit · x · 2026-09-09
Meta researchers propose MoEMB, scaling universal multimodal embeddings along the expert axis with mixture-of-experts instead of larger dimensions or reasoning tokens. MoE preserves single-vector, non-autoregressive encoding while sidestepping the contrastive-learning batch-size tradeoff. With only 3B active parameters, MoEMB sets new SOTA on MMEB-V2 and MRMR among models trained on public MMEB-family data, surpassing reasoning-token (TTE) methods with >4x active parameters at far lower compute, plus the first comprehensive study of adaptive computation for MoE embedders.
Related event: Meta's MoEMB: 3B Activations Beat 4x Larger Embedding Models(2 posts)→
More from Multimodal
- Dense instance segmentation via Muse Wonder + SAM 3.1 on Meta's Model API — nikhilaravi · 2026-10-26
- LichtFeld Studio v0.5.4.3 ships community-built SteadySplats, POPSpa and Voxelization — janusch_patas · 2026-10-26
- One prompt to a 2-3 min narrated short: hands-on test of Antigravity video generation — AI_Andrew · 2026-10-26
- Creator makes 10,564-frame animated short entirely with Gemini toolchain — AI_Andrew · 2026-10-26
- FLUX.3 Klein step-distilled to 4-8 steps with CFG baked in — Fit-Structure-6489 · 2026-10-26
- ostris pre-training "Primal VAE": 1-channel latents match modern 8x VAEs via patching — ostrisai · 2026-10-26