Cloudflare lets sites block AI training crawlers while staying searchable
djfergus · hn · 2026-09-16
Cloudflare's new blog post introduces "accountable mixed-use crawlers," addressing the long-standing dilemma where robots.txt forces an all-or-nothing choice: allow crawlers for search traffic or block them entirely.
The new approach lets site owners stay discoverable in search while explicitly disallowing the same crawlers from using content for AI training, adding accountability requirements for mixed-use bots. Discussion is ongoing on Hacker News about feasibility and crawler compliance.
More from Infra
- OpenRouter crosses 10 quadrillion annualized token run rate, doubling every 10.5 weeks — deedydas · 2026-11-02
- Palmer Luckey asks: can you use datacenter RAM for things prohibited by the operator's terms of service? — avaitopiper · 2026-11-02
- Disable MTP for 20-30% faster local LLM inference on limited RAM — InternalMode8159 · 2026-11-02
- Kioxia up 216% in a year but down 68% from its June 2026 peak — zephyr_z9 · 2026-11-02
- Intel plunges 29%, worst day since the 1970s, wiping $12B off US government stake — emmanuelvivier · 2026-11-02
- Researcher hoards nearly 1TB of DRAM, refuses to sell a stick to data centers — kevinnbass · 2026-11-02