Anthropic Scrapes Site 35,000 Times Per Visitor, Webmaster Shares Firewall Rules
threepointone · x · 2026-08-08
A webmaster checking server logs was shocked to find that 99% of traffic came from bots. Anthropic's crawler scraped the site 35,000 times for every 1 real visitor sent. He has documented and shared the Cloudflare and firewall rules used to block such aggressive scraping.
More from Infra
- Cost to Reach GPT-4 Level Intelligence Plummets 214x in Under Two Years — McDonaghMatthew · 2026-08-08
- AI Devours Memory Supply: Soaring Prices for Phones and Laptops Hit Consumers — 创业邦 · 2026-08-08
- Diamond Giant Devalues as Lab-Grown Diamonds Find New Life Cooling Nvidia GPUs — 创业邦 · 2026-08-08
- Exploring TPU Inference: How Do Consumer ASICs Compare in Performance and UX? — misanthrophiccunt · 2026-08-08
- NVIDIA Partners with Firebird to Build AI Infrastructure in Armenia and Kazakhstan — nvidia · 2026-08-08
- Optimized SQLite Repo: 1.59x Geomean Speedup via 4 Core Optimizations — rohanpaul_ai · 2026-08-08