WebArena-Pro: a 300-task multimodal benchmark across 20 self-hosted apps for web agents
McGill-NLP · hf · 2026-11-16
McGill-NLP and collaborators release WebArena-Pro, a heterogeneous, multimodal, reproducible benchmark with 300 tasks across 20 self-hosted web apps in six domains. It treats audio and video as core observations alongside text and vision. Tasks are authored via an annotator interface, LLM-triaged, human-validated, and run in isolation for parallel reproducible evaluation. Best agents: Codex with GPT 6.1 Sol and Claude Code with Claude Opus 5.5 at 62.3%, OpenCode with Kimi 3.1 Pro at 60.7%, while the strongest GenericAgent configs (Gemini 3.1 Pro omni, Gemma4 31B) reach only 30.3%.
More from coding & agent
- Continual-ARC: lifelong ARC-AGI stream finds agent harness persistence rarely beats simple caching — unifyai · 2026-11-16
- Filmworkbench v0.1.2: one-prompt semi-automated short films with Claude and ComfyUI — Eastern-Ad9515 · 2026-11-16
- Stop prompting LLMs for JSON: PowerShell module Jev routes with decision models — dfinke · 2026-11-16
- llama.cpp offloads CI caches and runners to Hugging Face to survive 35 PRs/day — ggerganov · 2026-11-16
- I killed my 30-minute weekly sync with a decision log my agents read before every run — Humaux · 2026-11-16
- A video editor tested free MCP-connected AI editing tools: invideo, ChatCut, FableCut — lol_you_again · 2026-11-16