A Firewall Rule Against Scrapers Can Quietly Erase Your Pages From ChatGPT Answers
shashib · x · 2026-11-14
The post flags a common enterprise conflict: security teams add edge rules to block scrapers, while marketing wants public docs and pricing pages citable by AI assistants — so a single firewall rule can silently remove product pages from ChatGPT, Perplexity and Claude answers.
Key points:
- robots.txt is only a preference; enforcement now happens at the edge and via cryptographic bot verification.
- Crawler roles must be separated: bulk model-training crawlers vs. search indexers (OAI-SearchBot, PerplexityBot) vs. user-triggered assistants (ChatGPT-User, Claude-User) need different rules.
- Verify identity first, then grant permission, using cryptographic signatures instead of blanket blocking.
- CMS owners sit in the middle, managing publishing controls alongside structured agent integrations like WebMCP and MCP endpoints.
More from Infra
- NVIDIA and TSMC reportedly never signed a formal contract in nearly 30 years of partnership — IgorCarron · 2026-11-14
- SemiAnalysis: SpaceX Memphis data centers to hit 2.4 GW by end of 2026, from 0.4 GW in January — mark_k · 2026-11-14
- SemiAnalysis warns VCs may lure away NVIDIA execs with billions as AI chip gold rush heats up — elonmusk · 2026-11-14
- Tesla Optimus plus Starlink could put robots in remote sites before full autonomy arrives — jeasinema · 2026-11-14
- Running AI on e-waste GPUs: 2.5 hours to render a 15s video on dual RTX 2060s — DevGnoll · 2026-11-14
- Anyone actually running local LLMs on a DGX Station B300? Seeking real-world numbers — spaham · 2026-11-14