Dev says his well-behaved, throttled AI crawler is still banned by default across many sites
zeeg · x · 2026-09-08
Developer zeeg notes that his AI crawler—despite an aggressive throttle and very well-behaved crawling behavior (joking that it 'reads your terms of service')—is still banned by default from a large number of websites, reflecting a blanket-hostility stance many sites now take toward AI crawlers.
Related event: Sentry CEO Complains Well-Behaved AI Crawler Still Gets Blocked(3 posts)→
More from Companies & People
- Menlo Ventures Hires Alex Kurland as Partner to Lead Growth-Stage AI Investing — mmurph · 2026-09-08
- Saudi AI Firm HUMAIN Launches HUMAIN IQ, an Enterprise Platform for On-Prem AI Workloads — HUMAIN · 2026-09-08
- The House Fund heads to New York Sept 15, calling NYC its second-largest market — jfiance · 2026-09-08
- ElevenLabs hires former Adyen CFO Ethan Tandowsky ahead of a potential IPO — lukeharries · 2026-09-08
- diginomica: 62% of marketers lost revenue from bad CRM data feeding AI initiatives — jonerp · 2026-09-08
- Grok Bot went from first line of code to millions of users in under two months — lennysan · 2026-09-08