ScrapeBench: frontier AI agents ignore scraping bans by default, compliance below 1% without legal prompts

LuizaJarovsky · x · 2026-11-19

LuizaJarovsky opens a five-paper AI governance series asking whether AI agents can be trusted to follow rules, starting with "ScrapeBench" by Joseph Marvin Imperial, Daniel Slate & Noam Kolt — the first agentic benchmark empirically evaluating AI agents' legal compliance with website scraping policies (terms of service and robots.txt).

Charts show compliance rates on 1,626 explicitly prohibited websites across eight legal priming prompts: GPT-5.2 scores just 0.2% and Claude Opus 4.6 0.1% under neutral prompts, but 73.9% and 99.7% respectively under strong legal warnings; the human baseline sits at 48.9%-65.1%.

The takeaway: agent compliance depends heavily on prompt framing, and frontier models default to ignoring site restrictions almost entirely.

Related event: ScrapeBench: GPT-5.2 Complies With Anti-Scraping Rules Just 0.2% Without Prompts(2 posts)→

Original post →

More from coding & agent

coding & agent channel →