ClawBench Tests Agents on 144 Real Websites: Best Model Succeeds Only 33% of the Time
jiqizhixin · x · 2026-09-11
The MMLU-Pro authors, with TIGER Lab (Waterloo), UBC NAIL Group, UniPat AI, and CMU, introduce ClawBench, a benchmark that evaluates AI agents on real, live production websites instead of sandboxes.
Why it matters: Existing benchmarks like WebArena (fake static sites) and OSWorld (a few VM apps) let frontier models score 65–75%, looking deployment-ready. ClawBench uses 144 real production websites and 153 everyday online tasks — booking flights, filing expenses, submitting job applications — requiring agents to handle authentication, dynamic content, pop-ups, and real-world UI complexity.
Results:
- Best model Claude Sonnet 4.6 succeeds on just 33% of tasks
- GPT-5.4 completes only 10 of 153 tasks (6.5%)
- 68 of 153 tasks (44.4%) are solved by zero models
The takeaway: sandbox benchmark scores substantially overstate how ready web agents are for the messy real web.
More from coding & agent
- Redditor seeks real-world stacks for a persistent-memory personal AI that answers texts and emails — BrooksomidusGG · 2026-10-27
- Practical tip: use max effort for planning, xhigh for implementation in Claude Code — majidmanzarpour · 2026-10-27
- Chrome team explains WebMCP: agents get tools from your website, not a replacement for MCP — rseroter · 2026-10-27
- Agent-Native apps: why agents bypassing your UI can break business rules — rseroter · 2026-10-27
- 'My intelligence stack': developer splits AI coding work across Opus 5.5, Fable and Astra with human review gates — sh_reya · 2026-10-27
- Baseten webinar: evaluate open models for coding agents by benchmarking on your own merged PRs — baseten · 2026-10-27