ClaudeBot crawler observed regularly fetching sitemap files for URL discovery
gaganghotra_ · x · 2026-10-20
Developer @gaganghotra observed ClaudeBot accessing sitemap files at a consistent weekly frequency. He speculates ex-Google engineers at Anthropic brought over search-era best practices: checking sitemaps regularly for URL discovery rather than relying purely on link crawling.
Related event: ClaudeBot observed crawling sitemaps weekly, Claude-SearchBot does not(2 posts)→
More from coding & agent
- Dev turns his Goodreads shelf into a slick site you can rebuild with Claude — thisiskp_ · 2026-10-20
- Notion AI ships major upgrade: smarter Agents, Auto 90% cheaper, BYOK support — ivanhzhao · 2026-10-20
- iPhone Action Button + Grok voice + HeyGen MCP turns a sentence into an avatar video — HeyGen · 2026-10-20
- Vibe-coded game League of Spawn adds Arenas: 15-minute MOBA team fights — majidmanzarpour · 2026-10-20
- esp32sim-mcp: MCP server lets AI agents build and test ESP32 firmware in a local emulator — EmanueleStrazzullo · 2026-10-20
- GameSummoner one-line local install: prompt-to-game pipeline runs fully local on a 5090 — demonicpigg · 2026-10-20