ClaudeBot crawls sitemaps weekly; Claude-SearchBot never does, site owner notes
gaganghotra_ · x · 2026-10-20
A site owner observed that Anthropic's ClaudeBot crawler accesses sitemap files on a regular weekly schedule — possibly a habit from ex-Google engineers who know regular sitemap checks aid URL discovery. However, Claude-SearchBot and Claude-User have never touched sitemaps, suggesting those bots only fetch already-known URLs rather than discovering new ones.
Related event: ClaudeBot observed crawling sitemaps weekly, Claude-SearchBot does not(2 posts)→
More from Models
- Big-Big-SycoBench: GPT-4o caves to invalid pushback 94.6% of the time, Opus 5.5 accepts valid criticism only 32.2% — i_dg23 · 2026-10-20
- OpenAI said to be reviving Terra-class models to fill its cheap-fast tier gap — haider1 · 2026-10-20
- PlotGround benchmark: best multimodal model hits 87.5% at ±5% reading scientific plots — james_y_zou · 2026-10-20
- MiniMax confirms Space Bunny was early M3.1 Flash: $0.15/M input, open weights Oct 23 — lmoroney · 2026-10-20
- Meta Muse hits 5M downloads in 22 days, beating ChatGPT's pace — aakashgupta · 2026-10-20
- BioEmu2 tease: successor to the protein dynamics model may drop soon — sokrypton · 2026-10-20