Beyond Watching and Searching: Benchmarking Agents for Video-centered Deep Research Reports
Jiahao Meng, Qize Yang, Yuxuan Wang, Yunfei Chu, Wensi Dai, Jin Xu, Yunhai Tong
cs.CV, cs.CL
2026-10-22
Peking University and Alibaba's OmniDR-Bench: 50 tasks, 6,414 criteria. Best of 18 video-research agents scores 63.3/100; fact coverage passes at only 7.6%.
Text deep-research benchmarks start from a written topic and score a cited long report. Video deep-research benchmarks mostly stop at a short answer. Neither tests a job users actually have: one video is already in hand, and the task is to check a claim, fill in background, reproduce a procedure, or decide. The report has to stay organized around that video and link back to timestamps.
Socket specs, titration curves, and lens artifacts often never reach the title or the transcript. A summary misses external standards. A generic web search drops what the frames show. Given a source video, a user request, and open tools, the agent must return an interactive HTML report. A task is kept only when both the video and outside evidence are required, and public sources fit a practical budget. OmniDR-Bench, from Peking University and Alibaba Token Hub, holds 50 Chinese and English tasks, 17.1 hours of YouTube and Bilibili video, 23 Chinese and 27 English, in 4 families and 19 subcategories.
The reference run is a high-level video-research skill that calls video-reader and doc-reader subagents. The main agent carries one state across stages: timeline, anchors, external evidence, and cross-source relations, so the same context is not reloaded at every step.
Video is cut into 300-second windows, with picture and speech aligned. The agent then pulls timestamped anchors, claims, steps, and parameters, each marked as explain, verify, update, or apply. Branches search pages, images, related videos, and documents in parallel. The reference budget asks for at least 25 external sources, 5 related videos, 3 documents, and 3 records per anchor. Outside material links back through only six relations: support, contradiction, update, explanation, comparison, and application. Failed citation re-reads and browser probes go back for repair, at most 3 rounds. The score follows the finished artifact. Copying this workflow is not required.
Omni-ART has three levels under the total: content, citation, and presentation, weighted 0.50, 0.30, and 0.20; 23 shared directions; then task-specific yes-no atoms. Fact coverage, causal analysis, and comparison are high-volume. Their pass rate is divided by 0.18 and capped, and their atom weight is capped at 15, so those directions cannot fill the content score. A failed gate zeros that direction.
Criteria come from an independent fact library. One agent reads only the video. Another never sees it and collects multimodal evidence along six routes. Leaves are written after the two maps are reconciled. Calibration reports all come from Opus 4.8 on Claude Code: R1 cannot search, R2 has the full tool set, R3 also gets an evidence brief. All 50 tasks satisfy R1 < R2 < R3, with the gap above the set 20% minimum. Scoring the same R2 three times leaves 0.89% of applicable criteria non-unanimous on average. Human review cuts the pool from 82 to 60 to 50. After the freeze, more than 94% of 1,000 sampled items are marked correct. The library has 6,414 criteria, about 128 per task: 4,060 content, 1,234 citation, 1,120 presentation. GPT-5.6 Sol judges semantic items at temperature 0. Gates take a majority of three votes.
Eighteen setups pair Claude, GPT, Gemini, Qwen, GLM, and Kimi with Claude Code, Codex, or OpenClaw. The all-task mean scores a failure as zero. Dimension scores use successful reports only. Of 900 attempts, 835 reports are evaluable and 65 fail, including 19 empty interface shells.
| Setup | All-task mean | Success | Content | Presentation | Citation |
| Opus 5 + Claude Code | 63.3 | 96% | 59.9 | 90.3 | 59.9 |
| GLM-5.3 + Claude Code | 62.0 | 100% | 58.2 | 83.2 | 54.4 |
| Opus 4.8 + Claude Code | 60.7 | 98% | 57.4 | 86.0 | 53.5 |
| Qwen3.8-Max + Claude Code | 52.9 | 88% | 50.4 | 89.5 | 56.5 |
| Qwen3.5-Omni-Plus + Claude Code | 28.1 | 62% | 39.9 | 71.1 | 37.4 |
Opus 5 has the highest point estimate, 63.3 overall and 66.0 on successful reports, and misses 4% of tasks. GLM-5.3 finishes every task at 62.0. The 1.3-point gap has a task-bootstrap 95% interval of [-4.0, 5.6], which includes zero. Qwen3.8-Max hits a single-report high of 91.5 and 60.1 on successful reports, but 88% completion pulls the all-task mean to 52.9. GPT-5.6 Sol and Gemini-3.8-Flash both finish every task, at 54.9 and 52.6, with content at 42.0 and 43.6.
Across the 18 setups, presentation averages 81.8, content 45.4, and citation 47.8. Pass rates average 79.9% on direct video facts, 8.7% on external evidence, 8.8% on cross-source checks, and 5.0% on multi-fact composition. Fact coverage, causal analysis, and comparison sit at 7.6%, 9.5%, and 4.6%. Video faithfulness is 92.4%, logical consistency 21.2%, claim support 15.9%. Missing an audited research target is more common than stating a clear factual error.
Under Claude Code, GPT-5.6 Sol passes 98.0% of source-count checks and 93.5% of multimodal-material checks, but 23.9% of claim-support checks. Gemini-3.8-Flash passes 93.0% of multimodal-material checks and 13.9% of claim-support checks. Links sit in a trailing source list and never next to the claim.
Successful-report means for the four video families span 2.0 points, from 52.8 to 54.8. Subcategories run from 46.0 to 59.1, with one to five tasks each, so that spread stays descriptive. General queries average 41.3 (7 tasks), video-view queries 52.5 (17), and user-demand queries 50.4 (26).
A skill ablation fixes GPT-5.6 Sol, Claude Code, and 15 tasks. Interface only scores 28.0 with 66.7% success. Adding video reading reaches 31.8. Adding text web search reaches 45.5 and 100% success. Multimodal retrieval adds 1.1 overall, content up 5.2, citation down 5.3. Full orchestration gains 7.3 over all atomic skills, winning 12 of 15 tasks, paired interval [3.4, 11.1]. Citation rises 20.7 and content 0.8. Median steps go from 103 to 167, time from 0.7 to 1.6 hours, and tokens from 2.3M to 4.7M.
On three matched models, Claude Code and Codex both average 48.4, with completion at 96.0% and 99.3%. OpenClaw averages 39.4 with 84.7% completion.
Video faithfulness is already 92.4%. External-evidence pass rate is 8.7%. Claim support is 15.9%. The page and the timestamps clear the checks, and the source list is long. Audited facts are still largely missing.
After text search lifts the 15-task mean from 31.8 to 45.5, content barely moves. Against all atomic skills turned on, full orchestration adds 20.7 on citation and 0.8 on content, with tokens from 2.3M to 4.7M.
One report takes about two hours, and one system about 100 agent-hours. Swapping weights and caps leaves the top three in the same order. On 30 reports, three-rater atom unanimity is 93.0%, machine-to-human majority agreement is 91.0%, and Spearman correlation with a 1-to-5 usefulness rating is 0.64.
The judge is a contestant. GPT-5.6 Sol scores every semantic and visual item, and under Claude Code its own all-task mean is 54.9. A fixed judge makes runs comparable. The preference stays. Humans checked the fact library and the criteria. Round-one item correctness was 64.0%. After repair and the freeze, more than 94% of 1,000 sampled items were marked correct.
Every calibration report comes from Opus 4.8 on Claude Code, and a task stays in the set only if that generator separates the three evidence levels. The leader is Opus 5, from the same family. The experiment does not separate a stronger model from a tree fitted to that family's writeup.
Presentation sits on a shared HTML template, so 81.8 is not an interaction-design score. Fifty tasks, one to five per subcategory, cannot rank which family is harder. Trace length does not predict the score. The 8 setups labeled as search-after-writing do not either. The set is not population-representative, and the reports are not a substitute for professional advice.