WeMM-Embedding: WeChat Multi-Modal Embedding Technical Report
Junjie Zhou, Ke Mei, Lei Li, Tianyi Wang, Fengyun Rao, Jing Lyu
cs.CV, cs.CL, cs.IR
2026-08-25
WeMM-Embedding adapts Qwen3.5 into 2B/4B/9B universal multimodal embedders. The 2B model scores 77.9 on MMEB-v2, above the previous leading 8B open baseline; the 9B model reaches 80.6 and is deployed in WeChat search and recommendation.
Retrieval, recommendation, classification, and agents all want one vector space for text, images, videos, and documents. CLIP-style dual encoders do not naturally handle interleaved inputs, videos with transcripts, or composed queries. MLLMs can read those inputs, but they still need large-scale alignment and finer relevance training before they are useful embedders. Open 2B models such as Qwen3-VL-Embedding and DME already sit in the mid-70s on MMEB-v2, and scaling those recipes to 8B has not opened a large gap.
All three sizes sit on native-multimodal Qwen3.5. An <embedding> token is appended, last-layer hidden state is pooled and L2-normalized. Several embedding tokens can share one sequence, so a video plus ASR can emit a video-only vector and a joint vector in one pass. Matryoshka training makes every prefix from 64-d up to full width a valid embedding.
Training examples share one pair format: optional instruction, source, positive target, optional hard negatives, optional graded score. Stage 1 trains on several hundred million pairs: weak web pairs, captions, retrieval, classification rewritten as source-label, multimodal QA, and human graded relevance. Each batch stays inside one task so in-batch negatives share a candidate space. Near-duplicates are masked to cut false negatives on shared labels. Graded batches use a score-gap-weighted CoSENT loss.
Stage 2 is about one tenth as large. Semantic IDs from three-level residual k-means down-sample frequent codes. An MLLM filters mismatched pairs and repairs dirty alt-text, then hard negatives are mined. Selected tasks add reranker-score ranking; the 2B and 4B models also distill bidirectional KL from a frozen 9B teacher. The 9B model has no larger teacher, so several Stage-2 specialists are merged. Reranker supervision is kept only where it actually helps.
MMEB-v2, 78 datasets:
| Model | Size | Overall |
| Qwen3-VL-Embedding | 2B | 73.2 |
| DME-Small | 2B | 74.8 |
| WeMM-Embedding | 2B | 77.9 |
| WeMM-Embedding | 4B | 79.2 |
| WeMM-Embedding | 9B | 80.6 |
The 2B model is 4.7 above same-size Qwen3-VL-Embedding, 3.1 above DME-Small, and slightly above the previous leading 8B open baseline. The 9B model led the official board on 2026-08-24. On MMEB-v3, audio tasks are scored zero because the model has no audio; V3-All is 56.0 / 58.2 / 59.5 for 2B / 4B / 9B, with the best Text and Agent group scores in the table. On the 12-dataset cross-modal suite from the Gemini Embedding 2 report, averages are 79.8 / 80.8 / 81.7.
On 26 in-house WeChat tasks the 2B model averages 72.0 against an open baseline at 60.9, winning all five categories. Fourteen online A/B tests were positive and the model is live in Channels, Official Accounts, Moments, and e-commerce. Per-test lifts are not reported.
MRL on the 2B model keeps 98.7% of 2048-d image and video scores at 256-d, about 99% at 512-d. Visual documents lose more. In a small Stage-1 ablation, dropping task-consistent batching cuts 71.9 to 68.5. Stage 2 walks 75.7 to 77.9, with distillation adding 0.9 and a larger visual budget another 0.3.
Universal multimodal embeddings are already production infrastructure. A 2B model beating older 8B recipes means one Qwen backbone can serve images, video, documents, and interleaved queries without a dual encoder per modality. 256-d vectors that keep most of the quality are the part that actually changes indexes.
This is a technical report. The transferable pieces are the pair format, task-consistent negatives, duplicate masks, and distillation for small models. The 26-task board and 14 A/B tests show deployment, not a reproducible effect size.
No audio, so MMEB-v3 All is dragged by eleven zeroed tasks. The hundreds of millions of pairs are described at a high level, including licensing. In-house and online numbers have no public protocol and no reported A/B effect sizes. The 9B model is a merge of specialists, so data and merging are entangled. The leaderboard snapshot is 2026-08-24. The authors list omni-modal inputs and larger sizes as future work.