One site's 30-day AI crawler log: Apple tops 2.45k, OpenAI 2.26k
gaganghotra_ · x · 2026-10-18
A developer shared Cloudflare stats showing their personal site received 26k AI crawler requests over 30 days, with 14k allowed and 12k unsuccessful (10.39k HTTP 403s). Apple's Applebot led with 2.45k requests, followed by OpenAI (2.26k), Google (2.17k), Anthropic (2.01k), Microsoft (1.23k), Perplexity (1.22k), DuckDuckGo, Baidu, Mistral, and Common Crawl. The most-crawled path was a post about ChatGPT's own search index (642 successful requests), and Apple crawlers fetched 344.15 MB of video.
More from Infra
- Dev builds a playable game on Surface Pro Ultra using local Qwen 3.8 27B — draginol · 2026-10-18
- One-click AI-ready PC: Windows & Ubuntu AI requirements installer tutorial for beginners — CeFurkan · 2026-10-18
- Basalt adds NVFP4: Qwen3.8 Flash-Next on a single 5090 with 8,582 tok/s prefill — jesdga95 · 2026-10-18
- Turn Your Idle Gaming PC Into a Build and Data Farm for Your MacBook Over SSH — pvncher · 2026-10-18
- ConnectWise supply chain attack: one OAuth app may expose hundreds of M365 tenants — cyb3rops · 2026-10-18
- Drone Hauling Kioxia SSDs Hits 2.95 Petabits/s: Sneakernet Strikes Back — rickasaurus · 2026-10-18