ClawBench Tests Agents on 144 Real Websites: Best Model Succeeds Only 33% of the Time
jiqizhixin · x · 2026-09-11
The MMLU-Pro authors, with TIGER Lab (Waterloo), UBC NAIL Group, UniPat AI, and CMU, introduce ClawBench, a benchmark that evaluates AI agents on real, live production websites instead of sandboxes.
Why it matters: Existing benchmarks like WebArena (fake static sites) and OSWorld (a few VM apps) let frontier models score 65–75%, looking deployment-ready. ClawBench uses 144 real production websites and 153 everyday online tasks — booking flights, filing expenses, submitting job applications — requiring agents to handle authentication, dynamic content, pop-ups, and real-world UI complexity.
Results:
- Best model Claude Sonnet 4.6 succeeds on just 33% of tasks
- GPT-5.4 completes only 10 of 153 tasks (6.5%)
- 68 of 153 tasks (44.4%) are solved by zero models
The takeaway: sandbox benchmark scores substantially overstate how ready web agents are for the messy real web.
More from coding & agent
- Stanford's PaperFormalizer turns research papers into Lean-verified versions with author in the loop — StanfordUniversity · 2026-10-27
- Hugging Face TRL makes score centering + masked importance sampling default in async GRPO — _lewtun · 2026-10-27
- AutoResearch agent's weird training curve gets dug into instead of discarded — ZimingLiu11 · 2026-10-27
- Firecrawl offers 2.5x credits to a16z Speedrun founders, unveils $75M Series B — devdigest · 2026-10-27
- Chums: open-source Apple Watch app puts Claude Code and Codex on your wrist — brandon_galang · 2026-10-27
- Open-source Clairic Studio turns pasted text into typed data and working code in 5 languages — dfinke · 2026-10-27