Anthropic crawler IPs can be silently blocked by AbuseIPDB reputation filters
Ill-Pack-9105 · reddit · 2026-07-23
A site operator says one of Anthropic’s published crawler IPs was silently blocked because it was flagged 100% on AbuseIPDB, even though robots.txt was open and other AI crawlers could fetch the same docs.
The post explains that many CDN/WAF reputation blocklists are default-on, so a crawler can be denied before robots logic runs. In this case, Anthropic’s shared cloud IPs appear to have accumulated abuse reports, which can cause legitimate crawls to fail without obvious monitoring signals.
More from Infra
- Anthropic S-1: training infra edge cut per-task cost 40% and doubled revenue growth — BenBajarin · 2026-10-26
- Rewriting scaling law study: optimal synthetic data share rises from 10% to 32% with compute — heghbalz · 2026-10-26
- US states halt new data centers as 72% of Americans oppose local builds — YvesMulkers · 2026-10-26
- $5M Seed Round Can't Afford One DeepSeek Training Run: GPU-less AI Startup's Three Paths — FinanceYF5 · 2026-10-26
- HBM demand shifts: low-stack inference plans shelved, Rubin to go 12-high only — BenBajarin · 2026-10-26
- CircuitHub unveils The Grid, a factory-scale robot making PCBs in the US twice as fast as Shenzhen — garrytan · 2026-10-26