Cloudflare Now Scores Verified Bots on robots.txt Compliance and 429 Response
shashib · x · 2026-10-18
Cloudflare has started scoring Verified bots on whether their crawling behavior respects site-published preferences.
- Two initial signals: sustained robots.txt compliance and whether the bot slows down after receiving HTTP 429 rate-limit responses
- The system is live inside Cloudflare today, with plans to let customers decide how much the score affects traffic handling
- Makes robots.txt and rate-limit behavior formal inputs into crawler reputation, directly relevant to the content-vs-AI-crawler standoff
More from Infra
- Korea Chip Stocks Fall 7.57% as Investors Price the Gap Between AI Demand and Financing Capacity — shashib · 2026-10-18
- Three receipts in 14 days: Anthropic's $15B SpaceX compute deal fuels the 'last window' thesis — elonmusk · 2026-10-18
- AI-generated kernels were mocked 2 years ago as 'just a while loop' — the world has changed — bingxu_ · 2026-10-18
- ClickHouse aggregates 100B rows in 0.4s with parallel replicas auto-scaling — Abhishekcur · 2026-10-18
- Auction-based LLM routing paper finds fixed token pricing wastes 10%-71% in costs — _onionesque · 2026-10-18
- Running Ling 3.1 Flash (560B MoE, 25B Active) Across 4 DGX Sparks via llama.cpp RPC — BerryBlush_Rogers691 · 2026-10-18