Omni-Embed-Mini: a 0.9B embedder binds six modalities, text retrieval stays bit-identical

Omni-Embed-Mini: Binding Modalities Without Forgetting via Dense Distillation

Mohammed Irfan Kurpath, Jaseel Muhammad Kaithakkodan, Sahal Shaji Mullappilly, Ivan Laptev, Hisham Cholakkal

EMNLP 2026

cs.CV

2026-10-02

A frozen text backbone plus dense-caption self-distillation gives a 0.9B model six modalities with zero text regression, 2.7-9.5x smaller than open omni embedders.

What problem this solves

Extending a text embedding model to images or audio usually costs you text retrieval: joint contrastive training shifts the text geometry and MTEB scores drop. The omni embedders on the market answer with parameters, with e5-omni, LCO-Embedding-Omni, omni-embed-nemotron and BidirLM-Omni all sitting between 2.5B and 8.9B. The paper calls this the multimodal expansion dilemma: under current recipes, every added modality forces a choice between text regression and model bloat, and text is still the most critical modality for real search.

The authors' position is blunt: text should not be a co-learner, it should be an immovable anchor.

Method

The central trick needs no separate teacher model. Every media sample gets a dense caption, and the frozen text backbone embeds that caption itself; that embedding is the distillation target. Teacher and student share the same frozen weights, so they live in identical geometry and cosine alignment suffices, with no projection head.

• The text backbone (Qwen3-Embedding-0.6B) is never updated. In the 0.9B model the text inference path is bit-identical to the stock weights, so training structurally cannot regress text retrieval.
• Speech runs through Whisper and environmental audio through Dasheng, each emitting 128 tokens that are temporally interleaved; image, video, and visual documents share one ViT. All modality tokens are spliced into a single sequence through one causal transformer, and the EOS hidden state is the embedding.
• Only 68.32M parameters (7.3% of inference) are trained: projectors plus r=16 LoRA. Projectors train alone for the first 20% of steps before encoder LoRA switches on, on the logic that adapters trained against misaligned projections learn noise. The loss is a Matryoshka SigLIP contrastive term plus cosine self-distillation, with a background FAISS miner supplying hard negatives that sharpen as the encoder improves.
• Captions come from Qwen3-Omni-30B with the dataset's original caption pinned in the prompt as grounding; visual-document captions average 654 words, about 6x other modalities. Training uses 228K rows on 8 MI210 GPUs.

Results

The 0.9B model's weak spot is vision: 26.29 on image and 18.48 on video, traced to a ViT extracted from a 0.8B backbone and only 86K image and 36K video training rows. Swapping in the 2.3B vision-language backbone lifts image to 64.80 and video to 55.18, so vision-encoder capacity, not the alignment recipe, is the limiter.

The most informative ablation is caption density: replacing dense cascaded captions with the datasets' originals costs 14.06 on video and 8.60 on image. Unfreezing the backbone at 0.9B improves every media modality by 1.10 to 5.66 but drops text by 3.13; at 2.3B it degrades everything. Across three seeds the modality scores stay within 2 points, with text pinned at 49.57.

Why it matters

For teams building omni-modal retrieval this is a replicable recipe: take the strongest off-the-shelf text embedder, bolt new modalities on through projectors and LoRA, and take zero risk on the text side. The 0.9B footprint suits edge deployment, and Matryoshka truncation to 128 dims shrinks a 10M-item index from about 41 GB to about 5 GB. A frozen backbone also means a future stronger text embedder can be adopted by re-running only the projector and LoRA training, which the 0.6B-to-2.3B swap already demonstrates.

To be honest, the 0.9B model clearly trails the larger baselines on media retrieval; its selling point is the text guarantee and its size. The 2.3B variant is the one that competes head-to-head.

Limitations

The authors' own list is thorough. The shared space is bounded by frozen text geometry, so hard-to-verbalize signal (complex acoustic textures, abstract visual patterns, fine temporal cues) may not fit. Every distillation target comes from a single captioner, so its systematic bias is applied uniformly across the corpus rather than averaged out; captions are not factuality-filtered and no caption-perturbation study was run, so all the paper can report is that retrieval falls hard when caption quality falls. Video evaluation mean-pools 8 frames for every omni model, capping video scores. No existing benchmark probes any-to-any retrieval across all six modalities, which limits what the overall average means, and interleaved multi-modal queries are native to the architecture but not isolated experimentally.

One more thing to state plainly: zero regression is a structural guarantee of inheriting Qwen3-Embedding-0.6B, not a claim of producing text capability.

Terms

• Matryoshka representation learning (MRL): training so embedding prefixes stay useful; vectors can be truncated to 128/256/512 dims at query time to trade quality for storage.
• SigLIP: a pairwise sigmoid contrastive loss that skips batch-level softmax and works well at small batch sizes.
• Hard-negative mining: periodically re-embedding the corpus with the current model and feeding the confusable near-misses back as negatives.
• Catastrophic forgetting: old skills being overwritten while learning new ones; here, multimodal fine-tuning breaking text retrieval.
• LoRA: parameter-efficient fine-tuning that learns low-rank weight increments instead of full matrices.
• nDCG@10: retrieval metric scoring how highly relevant documents rank in the top 10.

Source

• arXiv 2610.02148v1
• PDF
• Hugging Face
• GitHub · https://github.com/k-m-irfan/Omni-Embed-Mini

What people are saying

• Omni-Embed-Mini: A 0.9B Embedder Adds Five Modalities Without Touching Text Weights _reachsumit

Related papers

• Discrete codes first: DeepMind scales autoregressive image models to coherent 256×256 generation
• BrickWright: a LEGO SDK and validator take agents from 9–69% to 100% physical validity
• Best of 22 VLMs hits 58.5% on AgentVQA as visual-state errors dominate
• LoRA on 33B MiniMax-H3 cuts missed entity-count changes from 34.7% to 21.3%
• Qwen3-VL-8B training lifts RewardData artifact Finding F1 from 3.38 to 35.42
• RTAformer gains 4.8 SGDET R@10 on transition frames over matched OED

All paper explainers