Learning from Failures: Retrieval-Centric CoT via Hard Negatives for Unified Multimodal Retrieval
Zelong Sun, Jun Wang, Kaicheng Yang, Tiancheng Gu, Ziyong Feng, Zhiwu Lu
cs.CV
2026-08-06
UniME-R1 stops generating CoT from the query alone; an adviser first retrieves candidates, diagnoses the embedder's confusion, and writes correction reasoning, reaching 69.9 with a 2B model.
Unified multimodal retrieval maps text, images, video, and documents into one representation space. Using a vision-language model directly as an encoder is efficient, but compressing raw input into one vector often drops the fine-grained cues that separate the target from similar candidates.
Recent Reasoner-Embedder methods try to fix this with chain-of-thought: generate a rationale, then encode that text into the query. The problem is the reasoning looks only at the query, never at what the retriever actually returned. It can explain what the query describes but not what the retriever confuses. It also injects redundant noise when the initial retrieval is already correct.
UniME-R1's central claim is that retrieval reasoning should be grounded in retrieval feedback, not the query. The framework has two roles. A dual-mode embedder first retrieves an initial top-k in discriminative mode. A retrieval-aware adviser then analyzes candidates one by one to find the discriminative cues the embedder confuses. It separates failure diagnosis from semantic correction: a <cotfocus> field distills the confused attributes, relations, or events, and a <cotanswer> turns them into a concise description that complements the original query. Together they form Retrieval-Centric CoT (RC-CoT).
Inference uses an adaptive rerank-or-retrieve strategy. The adviser judges whether the target is in the top-k: if yes, it reranks the candidates; if no, it appends RC-CoT to the query, re-embeds in generative mode, and re-retrieves over the full corpus.
Training has three parts: mining hard negatives to simulate realistic failures; jointly optimizing direct and RC-CoT-augmented retrieval; and aligning the adviser through supervised learning and retrieval-oriented reinforcement learning (GRPO, with four rewards: format, judge, rerank, retrieval). The embedder is Qwen3-VL-2B/4B with LoRA; the adviser is Qwen3-VL-4B.
UniME-R1 takes the best overall score at both scales on MMEB-V2.
| Method (scale) | Overall |
| Embed-RL (strongest baseline) | 66.8 |
| UniME-R1-2B | 69.9 |
| UniME-R1-4B | 70.3 |
The 2B model already beats every medium-size (4B-7B) baseline, so the gains come from the framework, not scale. Against the most direct Reasoner-Embedder baselines, the 2B model leads UME-R1 and TTE by 9.8 and 6.8 points. Consistent gains also appear on general retrieval tasks (Flickr30K, COCO, ShareGPT4V, Urban1K).
The most telling ablation asks whether retrieval feedback helps at all. Query-only CoT scores 65.6 overall, random candidates lift it only to 66.5, and RC-CoT based on the real top-k reaches 68.5, beating those by 2.9 and 2.0 points. Dropping the <cotfocus> failure-diagnosis field costs 0.6 points, so summarizing the confusion exposed by candidates is itself useful. Rerank and re-retrieve are complementary, and learned routing is optimal among learned strategies; but oracle routing reaches 72.2, leaving room in the routing decision.
It reframes retrieval reasoning from发散 around the query to convergence around the retrieval failure, and that shift is a real contribution. The gains are only a few points, but they are consistent across benchmarks and modalities, and a 2B model beats larger baselines, which suggests feedback-driven reasoning over hard negatives is the right direction rather than stacking parameters.
For anyone building retrieval systems, this is a directly borrowable design: write reasoning by looking at the retrieved results, not in isolation.
The cost is latency. Initial retrieval, per-candidate adviser analysis, and possibly a full re-retrieval make the pipeline far more expensive than encoding once. Routing target-in queries straight to rerank mitigates this, but the adviser is a separate 4B model, adding a component to deploy.
The 2B-to-4B gain is only 0.4 points, so returns are already diminishing. All training data comes from the MMEB-V2 training set, and out-of-domain generalization is tested only on zero-shot retrieval, not more extreme distribution shifts. The 2.3-point gap between oracle and learned routing shows the adviser is not yet accurate at deciding rerank versus re-retrieve.