Cloudflare will block AI training and agent bots by default, while keeping search crawlers alive
The Batch (Andrew Ng) · rss · 2026-07-24
Cloudflare introduced new AI traffic controls that block training and agent bots by default while still allowing search crawlers, plus a monetization system that could charge AI agents for data access.
Publishers can now control bot access separately for search, training, and agent use cases, with more granular rules for ads pages and a new BotBase database of verified bots. Cloudflare says the move is meant to preserve search traffic while giving publishers more leverage over AI crawling, but it also raises costs for smaller AI labs that rely on public web data.
More from Infra
- NVIDIA says Nemotron 3 Ultra hit 97.1% on agentic RTL chip-design tasks — NVIDIAAI · 2026-07-27
- Local Qwen models power a robot that tests 78 smartphones’ battery life — gappyvalley · 2026-07-27
- MiniBot 2.40 adds xAI, HF Studio and vLLM support with inline media tools — Creative-Type9411 · 2026-07-27
- Apple smart glasses, Nvidia-SK AI data center deal, and Ctrip’s RMB 5.179 billion fine headline a tech roundup — APPSO · 2026-07-27
- DeepSeek funding rumor, EU AI transparency rules and OpenAI agent incident make a packed AI news roundup — 创业邦 · 2026-07-27
- QuixiCore argues native quantized kernels beat dequant-then-generic execution — QuixiAI · 2026-07-27