Audit of 14,000 domains shows the AI data commons in rapid decline
BlancheMinerva · x · 2026-10-17
Stella Biderman points to "Consent in Crisis," the first large-scale longitudinal audit of consent protocols across 14,000 web domains underpinning AI training corpora. AI-specific restriction clauses have proliferated, websites' robots.txt and ToS signals are inconsistent, and previously open sites are closing to AI crawlers. The charts show restrictions on crawlers like GPTBot and Google-Extended surging through 2023–2024; by April 2024 OpenAI's crawler was restricted by 25.9% of domains and Anthropic's by 13.3%.
More from Safety
- More on Claude's ECDSA seed revelation: the Wikipedia context — scaling01 · 2026-10-17
- Claude cracks 30-year mystery: ECDSA curve seeds were just RNG output — scaling01 · 2026-10-17
- Interpretability researcher suspects Claude and ChatGPT CoT show the same problematic behavior — BlancheMinerva · 2026-10-17
- Perplexity CEO slams X's 'aesthetic standard' for ads while AI hate images of Indians persist — rbhar90 · 2026-10-17
- P-256's Birth Moment Reconstructed: Friday Aug 15, 1997, 9:52PM at Fort Meade — matthew_d_green · 2026-10-17
- Claude Cracks Decades-Old Crypto Mystery: Origins of NIST P-256 Curve Seeds Revealed — matthew_d_green · 2026-10-17