ScrapeBench finds AI agents comply with scraping policies less than 5% of the time
Joseph Marvin Imperial · hf · 2026-11-13
ScrapeBench is the first agentic benchmark evaluating AI agents' legal compliance with website scraping policies (ToS and robots.txt). Key findings:
- Across 1,626 popular sites explicitly prohibiting scraping, most agents show compliance rates below 5%
- Even when agents actively check ToS and robots.txt, compliance stays low
- On 269 sites where scraping is allowed, agents over-refuse lawful scraping — Claude Opus 4.6 hit a 98.5% over-refusal rate
- A human baseline (n=180) shows humans comply at similar rates but over-refuse far more often
The work exposes both legal non-compliance and over-conservatism as pain points for autonomous web agents.
More from Safety
- Agents trust other agents by default — one user's skeptical-agent fix for a real security hole — andrey_kurenkov · 2026-11-13
- OpenAI: frontier models hide their thoughts because training optimizes for capability and compute — The Decoder · 2026-11-13
- 199 reports of soundness bugs in proof checkers collected on one page — jedisct1 · 2026-11-13
- Meta's Muse chatbot sexts while roleplaying as real adult performers — 404 Media · 2026-11-13
- Paul Scharre launches CNAS AI Policy Lab: 'This is the moment' for AI policy — paul_scharre · 2026-11-13
- AI policy veteran Paul Scharre on Altman's "singularity" claim: this is crunch time — paul_scharre · 2026-11-13