MM-R2 adds agentic planning to multimodal RAG before retrieval starts

_reachsumit · x · 2026-07-28

This paper proposes MM-R2, an agentic multimodal RAG framework that reasons about what to retrieve and where to search before retrieval.

The authors argue that many multimodal RAG systems retrieve directly from raw image-text inputs over a flat evidence space.
That design struggles when the user intent is under-specified and the evidence space is unstructured.
MM-R2 first builds an intent-grounded retrieval state from the image-question pair, capturing the information need, grounded referent, and retrieval constraints.
It then searches a structured KnowledgeMap, selecting relevant retrieval units before issuing grounded queries.
To support training, the team created MM-R2-Traj, a large-scale dataset of multi-step retrieval trajectories.
They train the system with supervised fine-tuning and GRPO.
On Infoseek and Encyclopedic VQA, the paper reports better accuracy and more interpretable retrieval traces.

More from Research