WebArena-Pro: A Heterogeneous, Multimodal, Reproducible Benchmark for Web Agents
Imene Kerboua, Fatemeh Pesaran Zadeh, Xing Han Lù, Weijian Qi, Alexander Miller, Junyi Song, Yunjia Tian, Dongjin Kang, Seyeon Choi, Marzia Nouri, Ewen Gueguen, Matteo Boglioni, Fengyuan Liu, Zeyi Liao, Mengqi Yuan, Yue Li, Alexandre Lacoste, Alexandre Drouin, Spandana Gella, Huan Sun, Gunhee Kim, Siva Reddy
NeurIPS 2026 E&D Track
cs.AI, cs.CL, cs.LG
2026-11-13
WebArena-Pro tests web agents on 300 human-validated tasks across 20 self-hosted apps and treats audio/video as core observations; the best agents solve only 62.3%.
Evaluation for web agents is stuck between two gaps. Self-hosted benchmarks (WebArena, VisualWebArena, WorkArena) are reproducible but narrow: WebArena covers 6 sites, and all 19,912 WorkArena task instances come from a single ServiceNow deployment. Modality stops at text and screenshots, so audio and video embedded in pages go untested. Live benchmarks on production websites cover more ground, but pages and data drift between runs, which makes scores incomparable over time. WebArena-Pro closes both gaps at once: 20 self-hosted applications, with audio and video promoted to core observations.
Twenty applications run as Docker containers, grouped into six domains (productivity, communication, analytics, business operations, design and media, travel). Each app reproduces the interaction pattern of a familiar service: NextCloud mirrors Google Drive, Penpot mirrors Figma, YT-Lite mirrors YouTube. Apps within a domain do similar jobs but differ in REST APIs, permission models, and DOM structure, so the benchmark tests transfer across interface conventions rather than memorization of one product. Container state resets to a seeded baseline after every task, keeping tasks independent and enabling parallel, reproducible evaluation.
Each of the 300 tasks has its own dedicated template, unlike WebArena's 241 templates instantiated into 812 tasks. Tasks split along two axes: 204 single-site vs 96 cross-site, and 221 text/vision vs 79 audio/video.
The audio-video integration is what separates this from prior work. Media sources appear as nodes in the accessibility tree; when the agent decides it needs them, it calls transcribe() for a local-Whisper transcript or extractframes() to sample video frames. Transcripts merge back into the tree as synthetic nodes and frames return as images, so temporal media enters the same observation structure the agent already uses. Omni models also get listen() and watch() for direct audio and video input.
Scoring uses a rubric-based verifier: each task carries human-written criteria tagged action or outcome, organized as a DAG where a node fails if its upstream failed. Graders run in order of determinism: backend-state checks, URL and action heuristics, then a Qwen 3.6 27B LLM judge as final arbiter. Tasks were authored in a dedicated annotator interface, screened by LLM triage that discards anything an agent solves on a first attempt, then human-validated before release.
Success rates at a 50-step budget:
| Agent configuration | Overall SR |
| Codex + GPT 6.1 Sol | 62.3% |
| Claude Code + Claude Opus 5.5 | 62.3% |
| OpenCode + Kimi 3.1 Pro | 60.7% |
| OpenCode + Qwen 3.8 Max Prime | 55.7% |
| GenericAgent + Gemma4 31B | 30.3% |
| GenericAgent + Gemini 3.1 Pro omni | 30.3% |
| GenericAgent + Qwen 3.5 9B | 5.0% |
By domain, communication is easiest (Codex reaches 81.8%) and travel hardest (best result 57.7%). Cross-site tasks are consistently harder: Codex drops from 67.2% to 52.1%, and Claude Code from 65.7% to 55.2%, the smallest gap among the five coding-agent configurations at 10.5 points. Gemma4 31B falls hardest, from 38.2% to 13.5%.
The modality split has a counterintuitive side: GenericAgent configurations score higher on audio/video tasks than on text/vision ones, which the paper partly attributes to transcribe() and extractframes() converting temporal media into the language and image inputs these models handle well. Coding agents show the opposite pattern: Codex and Claude Code both hit 57.0% on audio/video versus 64.3% on text/vision.
A media-tool ablation on 43 tasks that cannot be solved from titles or thumbnails: removing the tools zeroes Gemini 3.1 Pro and Qwen 3.6 27B on audio tasks, leaves Opus 5.5 at 14.0%, and restoring the tools lifts it to 48.8%. A full 300-task run costs between $166 (Codex) and roughly $800 (GenericAgent with Opus 4.7) in API spend; judging costs under $4 per run.
Among self-hosted, human-curated benchmarks this is the widest application coverage and the fullest modality support to date. The rubric verifier emits a per-task reward signal, and the paper notes the infrastructure can double as a training environment for RL or rejection sampling; container isolation also suits robustness experiments such as UI drift and prompt injection.
Two takeaways for practitioners. Cross-application coordination remains a real weakness: near-passing agents lose on the order of ten points once a task spans two sites, so workflow-automation evaluations should be discounted for cross-site settings. And the harness matters as much as the model: Gemini 3.1 Pro scores 48.0% under OpenCode while its omni variant manages 30.3% under GenericAgent, though the paper itself cautions that harness and tooling differences are tangled in such comparisons.
The authors flag two. The action space covers only prerecorded audio segments and discrete frames; live voice and streaming video are left to future work. And the leaderboard conflates model, harness, and tool configuration: a shared 50-step budget does not equalize token use, observation frequency, or latency, and costs vary with provider pricing and cache, so the 62.3% vs 60.7% gap should not be read as a model ranking.
Two more concerns from reading the paper. Per-domain samples are thin: analytics has only 16 tasks, so each task is worth 6.25 points and domain-level numbers carry real noise. The LLM judge is the final arbiter for rubrics that backend state cannot verify; the appendix reports its agreement with human annotations, but the paper does not quantify how judge errors propagate into overall success rates.