Cloudflare: AI agent traffic up 1,700% in a year, over half of web traffic now non-human
Cloudflare Blog · rss · 2026-09-30
Cloudflare argues the internet now has a "second audience" — AI agents — and is building ways for sites to charge them. Key points:
- Traffic shift: HTTP requests on Cloudflare grew from 63M/sec (end of 2024) to 115M/sec (peaks above 150M); daily AI agent requests grew 1,700%+ in a year, and for the first time over half of internet traffic isn't human.
- Broken bargain: AI answer engines strip traffic from sites — human traffic fell up to 40% in a year for heavily crawled categories (retail, software, finance); AI-training crawler requests rose from 22% to 52% of crawler traffic.
- Agents differ from crawlers: they act for real people (booking, comparing, buying) and scale with questions asked, not content published.
- Identification & control: Web Bot Auth cryptographically verifies agent requests (OpenAI, Google, AWS sign theirs; 500B+ verified requests weekly); controls split into Search/Agent/Training; the new Disallow AI Training signal is honored by Apple, Google, and Microsoft.
- Getting paid: 50+ publisher-AI licensing deals exist but are all bespoke. Cloudflare's Pay Per Use charges verified buyers only when content is actually used, with usage reporting that shows publishers which queries surfaced their work.
More from AGI Musings
- AI won't take jobs, they'll just go from fake to extremely fake, researcher quips — kevinnbass · 2026-09-30
- "Stop confusing obedience with intelligence": a pithy take on AI — PierceLilholt · 2026-09-30
- Lehmann: AI music will never replace human music, just as nobody watches computers play chess — _akpiper · 2026-09-30
- The rise of the Forward Deployed Engineer: the new role AI is creating — Pavan_Belagatti · 2026-09-30
- Hot take: metered AI pricing pushes us back to reality, unlike ad-funded feeds — curious_vii · 2026-09-30
- Bill Gates: if a robot replaces a worker, why doesn't it pay into the pension fund? — _akpiper · 2026-09-30