WebArena-Pro: open-source benchmark with 400 tasks across 20 real websites
xhluca · x · 2026-11-17
The McGill NLP team released WebArena-Pro, a self-hosted, reproducible, heterogeneous multimodal benchmark for browser agents: 400 tasks across 20 popular websites (including Airbnb, Google Flights, and Dropbox — sites missing from WebArena-family benchmarks), split into 100 dev and 300 test tasks, with multimodal interactions and tiered DAG evaluation.
Eval code is already available with three integration paths: BrowserGym/AgentLab (supervised workers plus a study dashboard), Harbor, or an external agent connected via the browser/media MCP service. @imenelker also built a CLI that makes managing evals easy, with subcommands for pulling environments, starting them, and dashboards.
Related event: McGill NLP releases WebArena-Pro browser agent benchmark(4 posts)→
More from coding & agent
- OunceMCP 1.0 RC: Token Budgets and Safe Writes for MCP Servers, 58-85% Fewer Tokens — gentleGiant96 · 2026-11-17
- Solo Dev Shares 6 Hard-Won Lessons on Prompting Video With a Real Person's Face — pvdyck · 2026-11-17
- Who Can Even Read AI-Generated Code? Garbled Screenshot Sparks Debate — gethackteam · 2026-11-17
- Orion foundation model predicts gaze: finds 2x UX bugs, cuts computer-use latency 31% — Scobleizer · 2026-11-17
- Coding Agents May Deliver 100-1000% Productivity Shock to Software Next Year — moultano · 2026-11-17
- Alibaba launches Qwen4-Max: 1M context MoE model tops Chinese models on RSI-Exam — HuaxiuYaoML · 2026-11-17