WebArena-Pro: a 300-task multimodal benchmark across 20 self-hosted apps for web agents

McGill-NLP · hf · 2026-11-16

McGill-NLP and collaborators release WebArena-Pro, a heterogeneous, multimodal, reproducible benchmark with 300 tasks across 20 self-hosted web apps in six domains. It treats audio and video as core observations alongside text and vision. Tasks are authored via an annotator interface, LLM-triaged, human-validated, and run in isolation for parallel reproducible evaluation. Best agents: Codex with GPT 6.1 Sol and Claude Code with Claude Opus 5.5 at 62.3%, OpenCode with Kimi 3.1 Pro at 60.7%, while the strongest GenericAgent configs (Gemini 3.1 Pro omni, Gemma4 31B) reach only 30.3%.

Original post →

More from coding & agent

coding & agent channel →