DeepVoyager-VL: Incentivizing Vision-in-the-Loop Search for Long-Horizon Multimodal Agents
Huanyao Zhang, Jiepeng Zhou, Runhao Zhao, Yanzhe Shan, Jiaoyang Chen, Bowen Zhou, Bo Li, Fang Wang, Jialong Wu, Zhengwei Tao, Lang Mei, Xiaohan Yu, Liyan Liu, Chong Chen, Wentao Zhang
cs.CV, cs.AI
2026-08-03
DeepVoyager-VL lets mid-search image evidence drive later retrieval instead of confining vision to input or answer, reaching a 58.6 average across ten multimodal search benchmarks.
Multimodal large language models carry static parametric knowledge, so they struggle on knowledge-intensive, fast-changing open-world problems. Multimodal deep search has emerged to let models actively gather external evidence over multi-turn interaction.
But existing work uses vision only at the input or answer stage, ignoring its role in intermediate reasoning, and is not built for long-horizon interaction. Image evidence rarely drives later retrieval, capping interaction depth and reasoning span. The paper groups prior paradigms into three: entity substitution, front-loaded cropping, and answer-side grafting. All build the reasoning chain in text first, with images as garnish; a mid-search image seldom becomes a necessary condition for a later retrieval.
DeepVoyager-VL is a vision-in-the-loop long-horizon multimodal deep-search framework, in three parts.
Data synthesis. Entities and events are extracted from Wikipedia and news into records (participants, relation type, time, location); spatio-temporally co-occurring records form macro-events, repeatedly interacting entity pairs form relation bundles, yielding an event graph. Each node is enriched with images that store retrieval context (to relocate it) and visual-content evidence (grounded attributes, counts, text, objects, relations), with the retrieval context deliberately kept from leaking the answer.
Agent framework. The core is separating image discoverability from observability. Search returns only lightweight URL-and-caption references; when needed, FetchImage materializes an image and CropImage carves a reusable local observation into a visual register that persists across turns. Actions include text search, image search, reverse image search, web visit, fetch image, crop image, and code execution. Before any tool output enters context, a goal-conditioned summarizer keeps only goal-relevant conclusions and image references, so raw returns cannot bloat the context. Vision-in-the-loop is enforced by counterfactual necessity: a newly fetched image must resolve a variable a later step needs, or the query does not count as requiring vision.
Training. Supervised fine-tuning only, no reinforcement learning. Problems are stratified by difficulty, teacher trajectories are generated for the retained ones, the vision encoder and multimodal merger are frozen, and only the language backbone updates.
Across ten multimodal search benchmarks (MMSearch, SimpleVQA, LiveVQA, FVQA, BrowseComp-VL, MM-BrowseComp, and others):
| Model | Direct answer | Agentic workflow | DeepVoyager-VL |
| 8B average | 17.5 | 35.3 | 54.8 |
| 30B-A3B average | 20.7 | 40.7 | 58.6 |
The 8B variant leads on eight of ten benchmarks and the 30B on nine, among scale-matched open-source agents. Against proprietary models under the same workflow, the 30B variant trails GPT-5.5 (67.2) by about 8.6 points. The framework-level comparison holds too: plugged onto GPT-5, Gemini-2.5-Pro, and Qwen3-VL-30B, DeepVoyager-VL beats LMM-Searcher and other prior frameworks. Ablations show that removing the summarizer, image search, fetch, or crop each drops the average by 2.5 to 4.0 points; and its visual-tool calls make up 64.3% of all tool use, against 10% to 40% for the compared methods, confirming vision-in-the-loop is real and not a label.
For anyone building agents or retrieval-augmented systems, this promotes vision from input-or-output garnish to a first-class citizen inside the reasoning chain, and uses counterfactual necessity to force the model to actually use images rather than paste one in. The data-synthesis recipe, an event graph plus a counterfactual constraint that makes vision indispensable, should transfer to building training data for other multi-hop-evidence agents.
The authors concede the system is built for answer-oriented search and does not evaluate full research workflows like open-ended decomposition, hypothesis iteration, reconciling conflicting sources, or long-form synthesis. The goal-conditioned summarizer calls an auxiliary vision-language model every turn, adding latency and deployment cost, and the paper does not characterize the trade-off among summarization quality, context compression, latency, and money. The visual working memory is cumulative and trajectory-local, with no policy for what to retain, consolidate, or evict, so redundant observations pile up over long interactions. One open question: vision-in-the-loop necessity is guaranteed by the synthetic data's counterfactual structure, but how often that "must-see" structure holds for real queries is not shown.