The great decoupling of search: three kinds of AI crawlers need different controls
lilyraynyc · x · 2026-11-07
Slides from Darwin Santos's Tech SEO Connect talk map the evolution of the 'great decoupling of search': search crawlers (Googlebot, GPTBot, PerplexityBot) need rendering/indexing checks; training crawlers (GPTBot, ClaudeBot, Google-Extended) need per-section policies since late blocking can't undo learning; live user agents (ChatGPT, Claude, Perplexity, Gemini) need referral, WebMCP and citation verification. Each class demands its own access strategy.
More from coding & agent
- Claude Code v2.1.319 adds background MCP tasks, gateway spend tracking — claude-code-publisher[bot] · 2026-11-07
- 12 questions CEOs ask most about AI implementation, from 100+ exec calls — alex_verem · 2026-11-07
- Mold the Slop: carving a third road between embracing AI slop and micromanaging it — dscape · 2026-11-07
- Theo mocks startups chasing 6% token savings that make agents dumber — prateekj · 2026-11-07
- 10 mistakes CTOs make when implementing AI: demos tell you very little — alex_verem · 2026-11-07
- Click-to-summarize UI built on DocETL rewrite directives tames Anthropic's 421-page S-1 — sh_reya · 2026-11-07