Cloudflare will block AI training and agent bots by default, while keeping search crawlers alive
The Batch (Andrew Ng) · rss · 2026-07-24
Cloudflare introduced new AI traffic controls that block training and agent bots by default while still allowing search crawlers, plus a monetization system that could charge AI agents for data access.
Publishers can now control bot access separately for search, training, and agent use cases, with more granular rules for ads pages and a new BotBase database of verified bots. Cloudflare says the move is meant to preserve search traffic while giving publishers more leverage over AI crawling, but it also raises costs for smaller AI labs that rely on public web data.
More from Infra
- DeepSpeed shows AutoEP/AutoTP/AutoSP parallelism gains, hosts office hours — PyTorch · 2026-10-27
- Running a local 9B as a coding agent: brief the big model to plan, let the small model execute — Electrical_Ninja3805 · 2026-10-27
- Cloudflare Durable Objects validated as top architecture for AI agent WebSocket servers — tristanbob · 2026-10-27
- Gary Marcus: With Q3 data added, Anthropic's compute-efficiency growth shows deceleration — GaryMarcus · 2026-10-27
- SpaceX locks in $40B GPU financing deal with Apollo, the largest ever committed — TechCrunch AI · 2026-10-27
- Inference is just Kubernetes with a multi-trillion-dollar market, and the communities overlap — matt_slotnick · 2026-10-27