WebArena-Pro: open-source benchmark with 400 tasks across 20 real websites

xhluca · x · 2026-11-17

The McGill NLP team released WebArena-Pro, a self-hosted, reproducible, heterogeneous multimodal benchmark for browser agents: 400 tasks across 20 popular websites (including Airbnb, Google Flights, and Dropbox — sites missing from WebArena-family benchmarks), split into 100 dev and 300 test tasks, with multimodal interactions and tiered DAG evaluation.

Eval code is already available with three integration paths: BrowserGym/AgentLab (supervised workers plus a study dashboard), Harbor, or an external agent connected via the browser/media MCP service. @imenelker also built a CLI that makes managing evals easy, with subcommands for pulling environments, starting them, and dashboards.

Related event: McGill NLP releases WebArena-Pro browser agent benchmark(4 posts)→

Original post →

More from coding & agent

coding & agent channel →