OpenAI, Google, Microsoft, Anthropic back new robots.txt standard to opt out of AI scraping
The Verge AI · rss · 2026-10-31
OpenAI, Google, Microsoft, and Anthropic say they will adopt a new standard letting publishers opt out of AI scraping. A shared "policy token" signal, proposed by Anthropic standards lead Kevin Kelley via the IETF, would let websites declare AI training preferences once across multiple AI companies.\n\nThe token lives within robots.txt, the long-standing mechanism for signaling which crawlers may scrape a site.
More from Safety
- Bug bounty researcher flags AI security platform sessions as potential training data of unpatched exploits — npinto · 2026-10-31
- Delaware draft law would let AI agents run companies, sign contracts and sue — 2C_ornot2C · 2026-10-31
- Grok says rumored Gemini 4 Carbon cyber model is wrapped in guardrails, not leaked carelessly — AccBalanced · 2026-10-31
- Alleged 8TB "Claude Fable 5" weight leak from AWS S3 looks fabricated, teardown finds — AccBalanced · 2026-10-31
- Claude Duet's stop rules show approval fatigue is the hard part of proactive agents — shoumikgoswami · 2026-10-31
- OpenAI's MCP Review Rules: Every Tool Needs Annotations, but Hints Aren't Security — shoumikgoswami · 2026-10-31