Reason Before You Retrieve: Agentic Planning for Multi-modal RAG
Tianyu Yang, Shir Simon, Zhenzhen Li, Minhao Cheng, Xiangliang Zhang
cs.AI, cs.CV
2026-06-24
Most multimodal RAG collapses image and text into one query before retrieval, which drifts. MM-R2 first structures what to look for, which object it maps to, and answer constraints, then routes over a clustered KnowledgeMap; it beats the strongest baseline on InfoSeek by nearly 8 points.
Multimodal RAG (mRAG) answers "look at this image, answer this question." Existing systems either retrieve text and image channels separately and fuse afterward, or collapse the whole image-question pair into a text query. Both drift, because what the question asks and which object in the image it targets are not specified before retrieval. The other problem is a flat search space: all evidence sits in one vector space under a single similarity score, so generic but loose content beats precise domain evidence.
MM-R2's pitch is to plan before retrieving. It turns each image-question pair into a structured "intent state" triplet: what knowledge to look for (information need), which object in the image it maps to (grounded referent), and what the answer must satisfy (type, granularity, scope). Once those three are fixed, retrieval has a target.
Retrieval no longer runs over one bucket of vectors. It runs over a structured KnowledgeMap: the corpus (6M Wikipedia passages from InfoSeek) is clustered with HDBSCAN into semantically coherent units, each given an LLM-generated title and description for routing. The agent acts in two steps, first selecting a unit, then issuing a grounded query inside it. "Where to search" and "what to search" are separated.
For training the authors built MM-R2-Traj, 900K multi-step trajectories (860K train, 40K validation, averaging 2949 tokens) each tagged with intent, toolcall, toolresponse, summary, and answer. The model trains in two stages, 5000 SFT steps then 90 GRPO steps, on Qwen2-7B.
Two benchmarks.
| Method | InfoSeek Unseen-Q | Unseen-E | All |
| Wiki-R1-7B | 47.8 | 42.3 | 44.1 |
| CoRe-MMRAG | 45.2 | 46.9 | 46.5 |
| MM-R2 | 54.1 | 56.0 | 54.3 |
MM-R2 beats the strongest baseline CoRe-MMRAG on InfoSeek by 6.3, 9.1, and 7.8 points. Across retrieval budgets, Top-1 is 47.9%, Top-2 is 51.0%, Top-5 is 54.3%. Routing itself is learned: Route Recall@1 rises from 0.23 to 0.74, so the model genuinely learns which unit to look in.
On Encyclopedic VQA MM-R2 reaches 39.4%, above most baselines but well below LLaVA-mR2AG's 55.1% (which uses Google Lens retrieval).
Turning retrieval intent from one fuzzy similarity match into explicit planning, fix the referent and constraints, then select a unit, then query, is the idea that travels to other agentic RAG setups. Interpretability comes along for free: the trajectory has structure, so errors split into routing mistakes versus query mistakes. For anyone doing visual QA with knowledge retrieval, "plan then retrieve" is more worth trying than stacking bigger retrievers.
The paper has no dedicated Limitations section. The authors concede the KnowledgeMap partition is not uniquely correct but an approximate routing scaffold, and that a query can span multiple units when boundaries are wrong. The bigger comparison gap: MM-R2 uses its own dense retriever over Wikipedia, while the strongest competitor LLaVA-mR2AG uses Google Lens, a commercial retriever, and beats it by 15 points on Enc-VQA. That is not an apples-to-apples comparison, so "which is better" cannot be cleanly answered here. The 900K trajectories are synthetic, and their quality depends on the teacher that generated them; the paper does not discuss bias along that data chain.