Cloudflare: bot traffic to surpass humans by May 2026, will pay sites agents crawl
Cloudflare Blog · rss · 2026-09-28
Cloudflare's 16th annual Founders' Letter argues AI is reshaping the internet:
- Web explosion: after plateauing from 2012–2025, new sites surged in mid-2025 — driven less by AI slop than by vibe-coding tools enabling non-coders to build; over 7 million developers now build on Cloudflare's developer platform.
- Bots overtake humans sooner: automated traffic was forecast to pass human traffic in H2 2027, but agent and AI-crawler growth pulled that forward to May 2026; at current trends, bot traffic will be 1,000x human traffic within five years.
- Tragedy of the commons: an agent recommending one lunch spot may crawl 1,000 restaurant menus, leaving 999 sites serving load with no payoff; over half of what good bots fetch is unchanged content.
- Risk to newcomers: agents favor whoever has the most information (usually incumbents), threatening consolidation. Cloudflare says it wants '500,000 AI companies, not five,' and is shipping crawler efficiency improvements plus a way for sites to get paid when agents use their content.
More from Companies & People
- CNBC's David Faber: Jensen Huang Not Behind Online Reinstatement Petition — firstadopter · 2026-11-16
- Microsoft launches free 18-episode 'Generative AI for Beginners' course — Aiden_Tech_Ai · 2026-11-16
- Ben Thompson: The Demise of Jensen Huang Marks the End of Nvidia's Prospects and the AI Bubble — morqon · 2026-11-16
- Engineer's Interview Prep Method: Use Claude to Find Which Answers Undersell Your AI Work — brandon_galang · 2026-11-16
- DeepSeek open-sources its app frontend "chat-demo", already at 3.9k stars — NielsRogge · 2026-11-16
- CNN meme: Jensen Huang's 'imperial control' reportedly extends to the Nvidia cafeteria brisket — kevinnbass · 2026-11-16