Cloudflare launches Bot Preference Sync to auto-sync robots.txt
Cloudflare Blog · rss · 2026-08-22
Cloudflare introduced Bot Preference Sync, a feature that automatically updates a site's robots.txt based on AI bot preferences (Search, Agent, Training) configured in the Cloudflare dashboard. Available to all customers, it ensures alignment between stated preferences and edge enforcement rules. Cloudflare also established transparency criteria for mixed-use crawlers: to retain search access when training is disallowed, crawlers must respect robots.txt, offer opt-out of AI summaries, and provide URL-level visibility into content usage.
More from Infra
- Economist: banning data centers is like banning new energy sources — Afinetheorem · 2026-08-22
- DSPy 3.3.1 Released: Hardened Python Interpreter and MCP 2.0 Support — dbreunig · 2026-08-22
- Training-free geometric KV routing cuts Qwen memory traffic by 16x — Electrical_Offer5667 · 2026-08-22
- Silicon Valley misunderstands the real reasons behind local opposition to data centers — jachiam0 · 2026-08-22
- Anthropic Hires OpenAI Hardware Veteran for Custom Silicon Team — eliebakouch · 2026-08-22
- Modal Releases Training Gym SDK to Simplify RL Post-Training Infrastructure — andersonbcdefg · 2026-08-22