Anthropic allows crawling while OpenAI and xAI block it
AaronBergman18 · x · 2026-08-19
A long-running sitemap monitor reveals significant differences in AI companies' crawler policies: Anthropic fetches without issues, while OpenAI's behavior appears suspicious and often fails. xAI has started to explicitly block vanilla HTML requests. This data provides a real-world glimpse into the varying openness and data access strategies of major AI labs.
More from Companies & People
- Five facts about Unitree founder Wang Xingxing: delayed graduation for XDog, quit DJI in 2 months — lxfater · 2026-08-19
- Anecdotes about Wang Xingxing: 90s Billionaire Who Failed English and Delayed Graduation for Robotics — lxfater · 2026-08-19
- AI didn't make design cheap, it made production cheap — oykun · 2026-08-19
- ChatGPT Ads to expand to 31 European countries for free tier users next week — kimmonismus · 2026-08-19
- Physical AI Creates New Jobs: Robot Fleet Operators, Sim Designers — Scobleizer · 2026-08-19
- AI Bubble: Math Reality and Capital Games — Complex_Commission22 · 2026-08-19