Developer builds a page to monitor AI crawler behavior, data next week
sujingshen · x · 2026-11-23
Developer sujingshen has set up a page monitoring AI crawlers and their scraping behavior, and plans to share what the data shows next week. As AI crawlers aggressively harvest web content, this kind of tracking reveals how often different AI companies' bots hit a site and how they behave — useful reference for content site owners.
More from coding & agent
- Agent OS vision: MulticaAI dashboard shows Lead, Backend and QA agents collaborating on a work board — jiayuan_jy · 2026-11-23
- 'Back in my day we coded by hand': meme nails how fast AI changed coding — 0xsachi · 2026-11-23
- A2A protocol walkthrough: how triage and billing agents collaborate on a refund flow — adnan_hashmi · 2026-11-23
- Dev Migrates From Vercel to Cloudflare in 35 Minutes With Claude, Cutting Bill From $70 to $10 — aloncarmel · 2026-11-23
- How A2A protocol works for agent-to-agent delegation in Microsoft Foundry — adnan_hashmi · 2026-11-23
- Aiden Bai demos new dev tool that runs fully locally in the browser — aidenybai · 2026-11-23