OpenAI will honor shared robots.txt so websites can opt out of AI training
rhiever · reddit · 2026-11-02
OpenAI announced it will honor a shared robots.txt signal, letting website owners declare opt-outs of AI training data crawling via the widely adopted standard, instead of configuring crawler-specific directives. This lowers the barrier for sites to express their AI training preferences.
More from Safety
- CVSS 9.8 Linux kernel bug CVE-2026-98432 allows remote takeover; Ubuntu still unpatched — NathanpmYoung · 2026-11-02
- AI race-swapping used to fake real incidents into racist viral videos, TikToker reveals — Creative-Impress6293 · 2026-11-02
- NYT op-ed argues AI consciousness is a near-term research and policy problem, not sci-fi — PeterBowdenLive · 2026-11-02
- $6.3B drained via one engineer's hacked laptop at Binance, FBI points to North Korea — LexSokolin · 2026-11-02
- Dev claims Amazon closed his account after modest compute purchases, sparking watchlist warnings — Kyrannio · 2026-11-02
- LiveOverflow jokes nobody asked him to post-train open models on exploit data — kingslyj · 2026-11-02