WebArena-Pro launches: 400 tasks across 20 real websites for browser agents
xhluca · x · 2026-11-17
The McGill NLP team released WebArena-Pro, addressing the gap that well-known benchmarks like WebArena, VisualWebArena, and WorkArena cover at most 5 websites, leaving out many common ones. It adds 20 popular sites including Airbnb, Google Flights, and Dropbox, spanning bookings, video, productivity, and more.
The benchmark includes 400 tasks (100 dev / 300 test), multimodal interactions, and tiered DAG evaluation, and is self-hostable and reproducible. Evals can run via BrowserGym/AgentLab, Harbor, or an external agent connected to a browser/media MCP service, with a CLI that makes managing evals easy.
Related event: McGill NLP releases WebArena-Pro browser agent benchmark(4 posts)→
More from coding & agent
- Cua and Scale AI benchmark computer-use agents: 97% non-intrusive on Omarchy vs 40% on Ubuntu — TianbaoX · 2026-11-17
- Perplexity launches Monitor API: scheduled web watching with webhook alerts at $1/1k runs — denisyarats · 2026-11-17
- Caltech Mathathon winners build agent swarm to formalize BPHZ theorem in Lean — ctjlewis · 2026-11-17
- Baseten launches Voice Agent Stack; Realtime Voice hits 73.5% on τ²-Bench at ~$0.01/min — baseten · 2026-11-17
- Dev building a self-hosted MCP gateway to centralize auth and deployment — hakku276 · 2026-11-17
- Hala MCP + Claude workflow generates pro-grade motion graphics videos from prompts — aziz4ai · 2026-11-17