AgentVidBench: A Multi-Hop Video Question Answering Benchmark for Evaluating MLLM Agents
Seoyeon An, Hyeonseo Jang, Minsu Kim, Chanho Lee, Younghan Park, Kangwook Lee
cs.CV, cs.AI
2026-09-18
KRAFTON's AgentVidBench poses 100 26-way multi-hop video questions; Gemini 2.5 Pro scores 51% single-turn and 58% with Video-ReAct.
Most video QA benchmarks still test a single pass: the model sees a clip, writes an answer, and we score the letter. Video-MME and EgoSchema live in that regime. People do not watch video that way. They scrub, zoom, reread a caption, and stitch two distant events into a cause.
The KRAFTON team starts from a known transformer limit: a fixed-depth, bounded-precision model cannot finish a hard k-hop chain in one forward pass. They turn that into a design rule. AgentVidBench only keeps questions that require active evidence gathering. The test is whether a system can hunt through a video, not whether it once saw the frames.
The set is small and expensive: 71 openly licensed videos, 100 multiple-choice questions. Clips run from 8 seconds to 27 minutes (median about 5.5 minutes) across 12 genres, from daily life and documentaries to sports and screen recordings. Each question has 26 options, A through Z, so chance is 3.85%. Annotators were barred from writing items solvable from one frame or one local timestamp.
Three annotators answer independently. Disagreement triggers a rewrite until a single verifiable answer remains. Then come 25 distractors: nearby integers for numeric answers, lookalikes visible in the video for text, plus traps such as "not mentioned." Difficulty splits into 13 easy, 28 medium, 29 hard, and 30 very hard. A typical item tags 3.75 fine-grained skills.
The distinctive piece is the milestone. Annotators first write a human investigation note, then compress it into tool-agnostic evidence steps: what must be seen, not which API to call. An LLM judge scores a predicted trajectory on five axes (task understanding, evidence coverage, grounding, completeness, reasoning faithfulness) and folds them into one trajectory score. Two agents can take different tool paths and still match if they collected the same evidence.
Their own baseline, Video-ReAct, is a ReAct loop capped at 20 steps and 15 tool calls. Tools are analyzevideo (time span, fps 1-20, resolution, focus region; Gemini 2.5 Pro as the shared viewer) and gettranscript (Whisper-large-v3). Comparators are STAR, DVD, and AVP, with only the planner swapped so tools stay fixed.
Single-turn, Gemini 2.5 Pro leads at 51% accuracy and 0.71 trajectory. GPT-5 sits at 29%, Claude Opus 4.7 at 30%. Open-source Gemma-4-31B reaches 38%; most Qwen3-VL and Qwen3.5 models land between 16% and 25%.
Agent wrappers split the field. STAR hurts almost everyone: GPT-5 falls from 29% to 14%, Gemini 2.5 Pro from 51% to 13%. Weak tools plus extra hops can erase what the backbone already saw.
Video-ReAct is the most consistent lift:
| Model | Single-turn | Video-ReAct |
| Gemini 2.5 Pro | 51% | 58% |
| Claude Opus 4.7 | 30% | 54% |
| GPT-5 | 29% | 49% |
| Qwen3.5-27B | 22% | 49% |
| Gemma-4-E4B | 16% | 51% |
AVP hits 56% with Gemini 2.5 Pro and the best token efficiency (accuracy per million tokens = 5.28). Video-ReAct is 3.76, single-turn 3.36. Highest accuracy is not cheapest. Dropping fps, resolution, or interval control cuts Video-ReAct from 58% to the 50%-56% band.
Video-agent work has been scored on questions a single pass can already solve. This set forces inspection, and it scores the trail of evidence separately from the final letter. The practical lesson from Video-ReAct is narrow: let the planner pick where to look and at what fps. A rigid plan-observe-reflect pipeline is weaker here.
One hundred items make a diagnostic, not a leaderboard.
The authors say the set cannot grow easily: open licenses plus hand-written multi-hop items and checked traces. Difficulty labels also mix human judgment with observed model scores, so the tiers are partly circular.
Trajectory scores come from an LLM judge. They argue MINERVA's MiRA punishes harmless differences in tool logs, which is why milestones ignore tools. That choice is not nailed down with large-scale human reannotation. Twenty-six options crush chance, but n=100 still leaves a wide interval. STAR's collapse may also be an implementation mismatch with the original paper, not a pure indictment of the framework.