Anthropic crawler IPs can be silently blocked by AbuseIPDB reputation filters
Ill-Pack-9105 · reddit · 2026-07-23
A site operator says one of Anthropic’s published crawler IPs was silently blocked because it was flagged 100% on AbuseIPDB, even though robots.txt was open and other AI crawlers could fetch the same docs.
The post explains that many CDN/WAF reputation blocklists are default-on, so a crawler can be denied before robots logic runs. In this case, Anthropic’s shared cloud IPs appear to have accumulated abuse reports, which can cause legitimate crawls to fail without obvious monitoring signals.
More from Infra
- A100 rental prices still rising in 2026 per a16z data, Jevons paradox hits compute — VraserX · 2026-10-27
- Anthropic S-1 reveals $800B+ in off-books future compute commitments — GaryMarcus · 2026-10-27
- ~42% of Anthropic's ~11GW contracted compute is NVIDIA GPUs — ns123abc · 2026-10-27
- EdgeRock-4B holds KV cache at 72 MiB across 256K context, 99.1% smaller than Qwen3.5-4B — alifcoder · 2026-10-27
- RTX 5080 price jumps from $1658 to $2318 in ten days on Amazon — Frizzy-MacDrizzle · 2026-10-27
- ComfyUI on 12GB VRAM: measured peaks show offloading turns 24s into 11 minutes — Impossible-Mail-4619 · 2026-10-27