OpenAI, Google, Microsoft, Anthropic back unified robots.txt signal to opt out of AI scraping
emmanuelvivier · x · 2026-10-31
OpenAI, Google, Microsoft, and Anthropic say they will adopt a shared "policy token" signal in robots.txt, proposed by Anthropic's Kevin Kelley via the IETF. Publishers will be able to reject AI training scrapes across multiple companies with a single rule instead of configuring each crawler separately. The move comes amid escalating publisher pushback, alongside the RSL pay-per-crawl standard and Cloudflare's default AI-crawler blocking.
More from Companies & People
- After 7,500 students: free AI curricula don't ship, accountability does — mdancho84 · 2026-10-31
- User spots purple-lit office tower, claims it's Nous Research HQ; Nous reposts — NousResearch · 2026-10-31
- Rumors say Google is killing AI Overview, citing ad click damage and compute strain — Dan_Jeffries1 · 2026-10-31
- Microsoft offers voluntary severance to ~60,000 employees while shielding AI teams — The Decoder · 2026-10-31
- India Should Build AI Chips, Not Another LLM, Argues Tech Commentator — PrasoonPratham · 2026-10-31
- Wired: Meta's Muse could upend many businesses — but hasn't yet — Wired AI · 2026-10-31