Anthropic, OpenAI, Google, Microsoft to back unified robots.txt opt-out for publishers
jonathanmendez · x · 2026-11-08
Per Nieman Lab, Anthropic, OpenAI, Google, and Microsoft will support a common robots.txt signal letting publishers opt out of AI scraping once instead of blocking each crawler individually.
- Proposed by Anthropic engineer Kevin Kelley via the IETF; Anthropic already supports it
- OpenAI says it will adopt the signal "in the coming months"; Google and Microsoft plan to follow next year
- Meta's comms VP Andy Stone called it "an important step," as Meta faces scrutiny over broad-scraping commercial terms it agreed to delay last week
Poster jonathanmendez responded sarcastically, telling publishers they are "now completely irrelevant."
More from Companies & People
- Google's across-the-board strength: frontier AI, cloud, robots—and it profits — vitaliychiley · 2026-11-08
- Amazon exec explains axed Sam Altman biopic as data centers sway voters in exit poll — adariostrange · 2026-11-08
- AGI House hosts AI & Policy dinner bringing builders face to face with diplomats — agihouse_org · 2026-11-08
- Security practitioner slams frontier labs' "tabula rasa" approach to cybersecurity — thedealdirector · 2026-11-08
- Musk on legacy: asked how he wants to be remembered, he answers "civilization" — and the Jose meme — elonmusk · 2026-11-08
- mitsuhiko's team flooded with applicants after hiring tweet — mitsuhiko · 2026-11-08