Applebot nearly matches Googlebot in weekly crawl logs as AI crawlers surge
gaganghotra_ · x · 2026-11-09
A webmaster shares 7-day crawler stats for his domain: Googlebot leads with 559 allowed requests, Applebot follows at 419, ahead of OpenAI's OAI-SearchBot (337), Amazonbot (321), BingBot (308), Claude-related bots (186) and PerplexityBot (88) — evidence AI crawlers are aggressively competing for web content.
More from Companies & People
- Andreessen: AI-native startups vs incumbents is 'widest gap' he's ever seen — ai · 2026-11-10
- Google rolls out broader AI chip lineup to challenge Nvidia in a market worth hundreds of billions — dinabass · 2026-11-10
- Thinking Machines engineer returns to Modal, reflecting on a year of frontier ML infrastructure work — josh_wills · 2026-11-10
- Nikita Bier goes viral joking that Jensen Huang's love life is impressively restrained — zacharynado · 2026-11-10
- Nomagic Team Headed to CoRL 2026 to Dig Into Physical AI Mastery — m_wulfmeier · 2026-11-10
- Palo Alto CEO Nikesh Arora's playbook for building a $268B company: sell more, acquire fast — davidsenra · 2026-11-10