Douyin Multimodal Embedding Model Technical Report
Haonan Chen, Chu Li, Zhicheng Wang, Yuanwei Liu, Yuanjiang Wang, Shaohua Jiang, Zhicheng Dou
cs.IR, cs.CL, cs.CV
2026-08-03
Douyin DME moves reasoning into a few latent tokens and keeps reconstruction training-only, hitting 74.8/78.4 (2B/9B) on MMEB-v2 with under 1 ms of added query latency.
Multimodal retrieval has to satisfy two requirements that pull against each other: it must serve efficiently under billion-scale indexing, and it must make fine-grained semantic distinctions for hard matches. Platforms like Douyin, Xiaohongshu, and YouTube are demanding on both ends. Existing routes each cover one side. Contrastive embedding models (CLIP, VLM2Vec) are efficient, offline-encodable, and fit industrial vector-search systems, but their supervision stays at the pair level ("this pair should be close, that one far"), which is too coarse. CoT-based models generate an explicit reasoning trace before emitting the embedding, which sharpens discrimination but is too slow to serve as the main retrieval encoder online.
The Douyin Search multimodal team (with Renmin University's Gaoling School of AI) targets exactly this dilemma: can fine-grained reasoning be folded into a standard bi-encoder to get CoT-level discrimination without paying for online generation?
DME is trained in two stages on a Qwen3.5 backbone, released at 2B and 9B.
Stage 1 is large-scale contrastive pre-training over 25 million query-document pairs covering text, images, video, visual documents, and mixed modality, using a single <emb> readout token to build a unified embedding space. Supervision is still pair-level; the goal is to lay the foundation.
Stage 2 is "semantic sufficiency" learning on 5 million higher-quality examples augmented with structured CoT supervision from a teacher model (Seed-2.0-Pro). The paper uses "semantic sufficiency" for a stronger requirement than pair-level alignment: an embedding should not only be close to relevant instances but grounded in retrieval-relevant evidence and should preserve fine-grained counterpart-side semantics (those of the matched query or document). Two mechanisms supply it.
One mechanism governs "how an embedding is formed" (look at the right evidence, organize it by role); the other governs "what an embedding must preserve" (it must be able to reconstruct the counterpart). The joint objective sums the contrastive loss with the two sub-objectives.
On the public MMEB-v2 benchmark, DME-2B scores 74.8 overall and DME-9B 78.4, both SOTA at comparable scale. Per modality (2B/9B): image 75.9/79.8, video 65.6/70.8, visual document 79.9/82.0. The video and visual-document gains are especially large; DME-9B hits 70.8 on video versus 67.1 for Qwen3-VL-Embedding-8B and 60.7 for TTE-v2-7B.
| Model | Size | MMEB-v2 overall |
| VLM2Vec-V2 | 2B | 59.2 |
| Qwen3-VL-Embedding | 2B | 73.2 |
| DME | 2B | 74.8 |
| Qwen3-VL-Embedding | 8B | 77.8 |
| TTE-v2 | 7B | 75.7 |
| DME | 9B | 78.4 |
An ablation turns the recipe on step by step: contrastive training directly on the Stage-2 data (no Stage-1 pre-training, no two mechanisms) is 70.9; adding large-scale Stage-1 pre-training reaches 72.5; adding latent reasoning 73.8; adding cross-conditional reconstruction 74.8. Stage-1 gains concentrate on video (55.3 to 59.3) and visual documents; latent reasoning gives the single largest boost on video (59.3 to 63.7).
The paper also introduces a fairly new quantification: representation completeness. Using one embedding as the only prefix condition, the model recovers the original tokens under teacher forcing. Self-directions (recovering a side's own input) hit 87.9% (q2q) and 74.3% (d2d) at Top-1; cross-directions (recovering the counterpart) reach 87.7% (d2q) and 65.9% (q2d). A single vector does carry most of the original tokens, not just a coarse retrieval summary.
In production, DME delivers a 2.92% relative gain on Douyin's in-house offline set (the four cross-modal directions range from +2.70% to +3.10%), and online A/B on Douyin search shows a 0.1% Lifetime (LT) gain. On latency, latent tokens add under 1 ms to text and video queries (about 0.8 ms per query at batch size 4) and are nearly negligible for image-text queries.
This gives industrial retrieval teams that want CoT-level discrimination but cannot afford CoT latency a reproducible path: compress reasoning into a few latent tokens, keep reconstruction supervision training-only, and serve a standard vector retriever. For people building RAG, multimodal search, or agent retrieval tools, it means fine-grained matching does not require reranking or online generation. The representation-completeness metric is useful on its own: it turns "is this embedding rich enough?" from intuition into a quantifiable diagnostic.
The authors do not include a limitations section; the conclusion only lists two future directions, data scaling and model-size scaling. A few points deserve discounting on a read-through. First, 0.1% LT is a real online gain but small, and 2.92% offline is solid rather than startling in production, so this reads more as engineering integration than a principle-level breakthrough. Second, at 9B the 78.4 leads Qwen3-VL-Embedding-8B (77.8) by only 0.6, and TTE-v2 uses a 76-task setting that drops two visual-document OOD sets, so the horizontal comparison is not clean. Third, representation completeness is measured under teacher forcing, which may overstate true recoverability, and the video cross-direction q2d is only 59.2%, reflecting a genuine information bottleneck when reconstructing video from a text query. Fourth, no weights or model card are released; it is a pure technical report, so reproducibility is limited.