Ovis-Embedding: Pushing the Frontiers of Universal Omni-Modal Embeddings
Ovis-Embedding Team
cs.AI
2026-09-21
Ovis-Embedding encodes text, image, video, and audio in one native backbone. Omni-3B scores 58.46 on MMEB-v3, 5.19 above the runner-up; VL-9B reaches 81.13 on MMEB-v2.
Retrieval is no longer a handful of modality pairs. A maintenance agent may take an abnormal sound plus a short note and need a video tutorial, a figure inside a PDF, or an old service record. Queries and candidates can mix modalities, and their scores have to be comparable in one index. That setting is any-to-any retrieval.
Most omni embedders bolt an audio tower onto a vision–language model, or align a separate audio encoder to a geometry that never saw acoustics. Fine-grained alignment then fights the original space. Ovis-Embedding starts from a native omni understanding model, where text, vision, audio, and video already share one Transformer.
Omni-3B initializes from Qwen2.5-Omni-3B, drops the speech Talker, and keeps the Thinker plus vision and audio front-ends. VL-2B and VL-9B initialize from Qwen3.5 for text, image, and video. Inputs are formatted with the native chat template; the embedding is the last non-padding final-layer state. There is no modality-specific projection head. Retrieval is cosine similarity.
Training pairs cover text, images and documents, video, audio, interleaved inputs, and agent tasks (tools, GUI, knowledge). The report cites about 50 million pairs and flags the figure as a placeholder for the camera-ready freeze.
Four stages. Stage-1 is LoRA contrastive pretraining with gathered negatives, a focal weight on unresolved queries, and distillation of a teacher's full similarity distribution. Full-parameter updates at this point scramble a still-chaotic embedding space. Stage-2 unfreezes the model and draws each micro-batch from one dataset, so in-batch negatives stay on-task. Stage-3 keeps teacher-correct, student-wrong items, upsamples them, and continues with forward KL. Stage-4 freezes the encoder and attaches a shared PCA basis plus a residual adapter so one 2048-d vector can be served at 1024, 512, 256, or 128.
Omni-3B scores 58.46 on MMEB-v3, against Tianmu-Emb-Uni at 53.27, e5-omni-7B at 47.14, and Omni-Embed-Nemotron-3B at 43.60. It leads every group: image 77.55, video 64.99, visual documents 78.26, text 47.15, audio 50.08, agent 45.52, with margins of 3.72, 5.62, 2.89, 3.53, 7.04, and 6.10 over the runner-up. Across 31 entries it is first on 22 and second on 8. MultiConIR is the only miss, about 7.94 behind Nemotron-3B.
| Benchmark | Ovis | Comparison |
| MMEB-v3 overall | 58.46 | Tianmu-Emb-Uni 53.27 |
| MAEB Mean(Task) | 57.29 | next best about 3.75 lower |
| MVEB Mean(Task) | 61.77 | LCO-Embedding-Omni-7B, 4.19 lower |
| RTEB 15-task English | 67.35 | Qwen3-Embedding-4B 67.27 |
| MMEB-v2 VL-9B | 81.13 | Octen-VL-Embedding-Large, 1.04 lower |
| MMEB-v2 VL-2B | 77.46 | Octen-VL-Embedding, 2.04 lower |
MAEB and MVEB ranks are estimated by inserting local scores into a leaderboard snapshot; Omni-3B had not been submitted. Video clustering trails the best result by 2.00; memory retrieval trails Nemotron-3B by 2.79. Truncating to 1024-d keeps 100.1% of the full-width average; 128-d keeps 93.2%, with the largest drops on visual documents and agent tasks.
The portable recipe is a native backbone, homogeneous negatives, focal plus distribution distillation, and post-hoc width adapters. A 3B omni model beats several 7B omni embedders and slightly tops a larger text-only Qwen3-Embedding-4B on RTEB, so modality coverage need not tax text retrieval. For anyone building one index, fewer towers and fewer score calibrations is the practical win.
The 50 million-pair count is a placeholder. MAEB/MVEB ranks are local insertions, not official submissions. MultiConIR, memory retrieval, and video clustering still lag. Short prefixes hurt VisDoc and agent tasks, where fine-grained directions live in the discarded variance. Training compute and data mix are barely reported. The VL line has no audio path. Baselines use official checkpoints and their own pooling rules, which is fair and also means there is no single extraction protocol behind the gaps.