Sites block AI crawlers yet expect AI search visibility, dev points out
gaganghotra_ · x · 2026-10-28
Developer Gagan Ghotra highlighted a contradiction: many companies want AI search visibility, but their firewalls return 404s or outright block fetches from AI companies' bots. A screenshot shows a site's Cloudflare security service blocking requests from AI-company crawlers. Unless they allow these fetches, such sites may never get cited in AI search results.
More from Fun
- Community challenge: can a single bit flip observably change an LLM's behavior? — burny_tech · 2026-10-28
- Researcher's Sharp Take: e/acc Is auDHD, EA Is Malignant Psychopathy — mjdramstead · 2026-10-28
- 'The free energy principle is free because it's not that useful' — researchers clash over FEP — mjdramstead · 2026-10-28
- Beff Jezos raves about Astra Ultrafast's token speed — beffjezos · 2026-10-28
- AI CEO memes about girlfriends leaving over Dyson swarm ambitions — beffjezos · 2026-10-28
- Users notice ChatGPT almost always ends replies with "One question:" — Angaisb_ · 2026-10-28