Video-DeepResearch: Towards the Next-Generation Multimodal Deepresearch Agent
Zhen Fang, Yu Zeng, Wenxuan Huang, Yiming Zhao, Shiting Huang, Tianfei Ren, Qi Lu, Qingnan Ren, Qisheng Su, Lionel Z. Wang, Qingyu Yin, Shuang Chen, Zehui Chen, Lin Chen, Zhenfei Yin, Yao Hu, Shaohui Lin, Wanli Ouyang, Shaosheng Cao, Feng Zhao
cs.CV, cs.AI
2026-08-05
Closed flagships fake watching video, answering from memory. Video-DR forces vision-first with staged tool unlocking; its 35B open model scores 64.0%, beating Claude-4.5-Sonnet (59.0%).
"Deep research" agents that orchestrate tools and web search work on image-text tasks, but moving them to continuous video has been hard. The authors diagnose two recurring failure modes.
First, modality bias: agents skip visual tools and fall back to text search. In the diagnostic study GPT-5 calls visual tools zero times on video tasks and text tools only 0.12 times; Qwen3.5-397B manages 0.10 visual and 1.27 text. The agents look multimodal but are not actually watching the video.
Second, parametric knowledge leakage: models answer from memorized parameters instead of calling tools to verify. Video QA is especially exposed because many questions are guessable from common sense.
Together these mean a model can be "right" without being right-about-the-video, which wrecks both generalization and trust.
Video-DeepResearch (Video-DR) forces the model back onto the video with two designs, under a "look enough first, then search the web" philosophy.
The first is a decoupled perception-exploration pipeline with stage-wise tool unlocking. Stage one opens only visual tools (SelectKeyframe, CropSearch), compelling the model to ground every relevant frame before anything else. Only after visual grounding is done does stage two unlock text tools (Search, Visit). The model cannot jump to text search and skip the frames.
The second is a data pipeline that actively removes questions answerable without tools. For each generated question, four tool-free rollouts are run; any question answered correctly bare gets discarded (parametric-leakage filtering). This leaves 30k video-grounded QA pairs and 7k correct tool-using trajectories.
Training is two-stage: SFT (7k trajectories plus 7k text QA, the latter chosen to fix under-use of text tools), then GRPO (2k moderate-difficulty questions, binary reward), which breaks through the imitation-learning ceiling.
The authors build VideoDR-Bench, 200 human-AI collaboratively annotated multi-hop video QA instances spanning knowledge, entertainment, daily life, game and sports, news and more, across short to long video lengths. Main results (VideoDR-Bench average accuracy):
| Model | Avg accuracy |
| GPT-5 | 52.5% |
| Gemini 2.5 Pro | 57.5% |
| Claude-4.5-Sonnet | 59.0% |
| Qwen3.5-397B-A13B (open base) | 52.8% |
| Video-DeepResearch-30B-A3B | 59.3% |
| Video-DeepResearch-35B-A3B | 64.0% |
The 35B model scores 64.0%, 5.0 points above Claude-4.5-Sonnet, leaving GPT-5 (52.5%) and Gemini 2.5 Pro (57.5%) behind, and reaches 68.0% on its own Video-DR benchmark. It is a 35B model with 3B active parameters, far smaller than the closed-source flagships it beats. The 30B variant hits 59.3%, already level with Claude-4.5-Sonnet.
Tool-calling behavior tells the story best: the base model calls visual tools 0.10 times, Video-DR raises that to 2.33. The model is actually watching. Ablation climbs from a 40.5% base to 59.3%, with SFT data volume and RL each contributing a slice.
Video is the hardest modality to land right now, and closed-source flagships largely fake watching it. This paper offers a reproducible open path: enforce visual-first by the order in which tools unlock, and close the memorize-and-answer shortcut with data filtering. A 35B open model then beats several closed flagships at video deep research. Data and code are open (Osilly/Vision-DeepResearch), giving video-agent teams a ready starting point.
Performance is bought with heavy GPU resources: large-scale concurrent model deployment plus live web search makes reproduction non-trivial. The benchmark depends on careful human annotation, which caps how fast the dataset can grow; 200 questions is small, and category scores vary widely (news 41.7% versus entertainment 65.9%), so the average can mask imbalance. GRPO downsamples formatting-violation negatives at 20% probability, an engineering trick whose robustness across settings still needs checking.