AgentVQA: A Unified Benchmark for Agentic Visual Understanding
Advait Gupta, Sachin Konan, Tom Yin, Gabriel Sarch, Zhuang Liu
cs.CV
2026-10-23
AgentVQA is 1,000 offline questions across five agent domains, drawn from 23 datasets. The best of 22 VLMs scores 58.5%, and most errors are visual-state failures.
VLMs already make decisions in web navigation, robotics, and games. Comparing those decisions is the awkward part. Online evaluation lets a model act inside a simulator or a real environment, but setups are brittle, runs are hard to reproduce, and sweeping many models is expensive. General suites such as MMMU, MMBench, and GPQA test knowledge and image understanding. Their questions do not look like the next step an agent has to take. Strong scores there also fail to predict agent success: GPT-4o does well on older benchmarks and still lands under 50% on multi-turn customer-service tasks.
Domain-specific agent benchmarks stay narrow, disagree on answer format, and are sometimes saturated or noisily labeled. A webpage click and a 3D spatial judgment are not on one scale. AgentVQA puts five domains into a single offline set, using frames taken from real interactive settings.
The benchmark has 1,000 four-way questions, 200 per domain, from 23 source datasets at 40 or 50 items each. The domains are GUI clicking and navigation, egocentric video extended over time, embodied perception and action, reading a game state, and 3D spatial relations and paths. Coordinates, actions, spatial relations, and predicted next states all become multiple choice, with the original question kept where possible.
Models draft and screen. People decide what remains.
About 22% of the initial pool survives. A box around scissors includes background, so a click on empty space still counts. Deleting a Lyft account can start by opening the app or by opening the drawer first, and the question never asks for the shortest path. Those items are removed or rewritten. That human pass is what catches grounding bugs the automatic checks let through.
Twenty-two open and closed VLMs are evaluated. Models without native video receive 32 evenly spaced frames. The paper says an ablation and prior work support that count; the main text does not report the curve. Reasoning models use temperature 0.8, other models 0.2, and GPT models use their default. The prompt asks for one letter. Coordinate choices follow each model's native layout: Gemini gets normalized [0, 1000] pairs in (y, x) order, Qwen gets absolute pixels. Images stay at native resolution.
Random guessing on a four-way item is about 25%. Gemini 3.7 Flash leads the 22 models at 58.5%. Gemini 3.1 Pro scores 55.7%. GPT-5.6-sol with thinking-high and Qwen3.7 Flash both score 55.0%. Llama 4 Maverick at 28.0% and GPT-4o at 31.4% stay close to chance.
| Model | Overall | Web | Egocentric | Games | Spatial | Robotics |
| Gemini 3.7 Flash | 58.5 | 40.5 | 67.0 | 70.0 | 62.5 | 52.5 |
| Gemini 3.1 Pro | 55.7 | 34.5 | 64.0 | 68.0 | 61.0 | 51.0 |
| GPT-5.6-sol think-high | 55.0 | 47.0 | 69.0 | 62.5 | 51.0 | 45.5 |
| Qwen3.7 Flash | 55.0 | 43.0 | 72.5 | 61.0 | 55.0 | 43.5 |
| Qwen3.8 27B | 50.9 | 30.0 | 69.0 | 61.5 | 51.5 | 42.5 |
| Claude Opus 4.8 | 44.0 | 35.5 | 60.0 | 50.5 | 41.0 | 33.0 |
| GPT-5.6-sol think-none | 38.2 | 16.0 | 59.0 | 44.5 | 41.0 | 30.5 |
| GPT-4o | 31.4 | 21.5 | 40.0 | 29.0 | 35.0 | 31.5 |
| Llama 4 Maverick | 28.0 | 18.5 | 49.0 | 23.0 | 30.0 | 19.5 |
The paper's cross-model picture is about 59% on egocentric video, 51.5% on games, 46% on spatial planning, 38% on robotics, and 29% on the web. Web is hardest: only three models clear 40%. Games spreads the widest, from 23% to 70%.
No model leads every domain. Gemini 3.7 Flash leads games at 70%, spatial planning at 62.5%, and robotics at 52.5%, then scores 40.5% on the web, behind GPT-5.6-sol thinking-high at 47%. Qwen3.7 Flash leads egocentric video at 72.5%.
The paper treats the Qwen3.7 series as closed. The best open model, Qwen3.8 27B, scores 50.9%, 7.6 points behind Gemini 3.7 Flash. Kimi K3 at 48.3% and Qwen3.5 35B at 47.9% still beat Claude Opus 4.8 at 44.0% and Sonnet 5 at 43.4%, and they beat GPT-5.6-sol with thinking off at 38.2%.
Longer reasoning is not a general gain. GPT-5.6-sol moves from 38.2% to 55.0%, up 16.8 points. GLM-4.1V-9B scores 35.4% with thinking and 36.3% without. The paper attributes the drop to thinking loops.
Action history is a steady help on web trajectories. Across Gemini 3.1 Pro, GPT-4o, Qwen3.5 27B, Kimi K3, and Claude Sonnet 5, dropping the history costs 9.5% on AitW and 10% on MONDAY.
Ranks do not follow general benchmarks. On subset reruns with one prompt, Spearman ρ is 0.67 versus MMMU-Pro (250 items), 0.59 versus GPQA-Diamond (150), and 0.44 versus ChartQA-Pro (250). Against GPQA-Diamond, no model keeps its relative rank. Agreement with the online benchmark SpatialWorld shows up in the matched domain: over 12 shared models, AgentVQA spatial planning versus the SpatialWorld physical track has ρ = 0.895 (p = 8×10^-5). Other domains correlate more weakly.
The error breakdown covers Gemini 3.1 Pro only. People set the categories, Gemini assigned them, and people checked 100 labels. Shares are among wrong answers inside each domain. Web errors are 78% grounding, a miss on the right interface element. Games are 71% misreads of the board. Robotics is 65% spatial confusion, and spatial planning is 62% viewpoint failure. Egocentric errors split into motion misperception at 39%, perception gaps at 29%, and spatial confusion at 15%. Missing the task, or answering against the model's own trace, is rare in every domain.
MMMU and GPQA ranks are a poor shopping list for an agent VLM. Correlation with AgentVQA is only 0.44 to 0.67. The spatial-planning slice tracks an online eval at ρ = 0.895, without standing up a simulator, so it can separate models before a real-environment run.
The misses happen while looking, not while parsing the request. Controls, motion, state changes, and 3D occlusion dominate. Thinking and action history both help, and neither is enough. Thinking adds 16.8 points for GPT-5.6-sol and a small loss for GLM-4.1V. History buys back 9.5% on AitW and 10% on MONDAY. The region above 58.5% stays open.
An average flattens the split. The same cohort can sit near 70% on games and near 40% on the web. The best open model is 7.6 points off the closed leader, so the useful number is the domain score.
There is no new model here. The gain is a shared multiple-choice format with hard negatives across five domains, plus rank agreement with online evaluation where the domain matches. Code and data are public.
Online interaction remains the standard. The paper never checks whether an offline error is the failure a live rollout would hit, or whether a higher multiple-choice score becomes steadier online behavior. The ρ of 0.895 covers only spatial planning against SpatialWorld's physical track. Other domains agree less. History and longer reasoning leave the gap in place.
Error shares come from a single model. Gemini helped label them, and humans reviewed 100 assignments, so 78% and 71% do not transfer to GPT or Qwen. Distractors were written by Gemini 2.5 Pro, which can bend what counts as hard toward that family.
Four-way choice is looser than emitting an action. The model only has to pick. A deployed system has to write the coordinate or the action. Option numbers are also rewritten per model, so Gemini's (y, x) pairs and Qwen's pixels are not the same question. Items with several valid paths were removed. The yardstick is cleaner, and further from a real branch. Only about 22% of candidates remain, and easy items were thrown out by hand. The main table has no multi-seed runs and no intervals. At temperature 0.8, one run can shuffle the middle. The 32-frame ablation is not quantified in the main text.