AI scrapers outnumber humans 20:1 on a personal archive site, only 0.7% of 4.3M requests are human
Ok-Bullfrog-3052 · reddit · 2026-11-22
A webmaster publishing 35 years of historical data found AI training scrapers have overwhelmed his site: over 30 days, humans accounted for just 0.7% of 4.3M requests; scrapers outnumbered humans 20:1, and every newly published file is instantly grabbed by at least 10 bots.
Key observations:
- The 50GB daily cap is exhausted by 3am, locking humans out of the original images; after a Facebook announcement, Meta started indexing within 1 second, other agents within a minute
- More than half of requests were errors from misaligned agents and misconfigured legacy tools, most failing to fetch even one file; 257,000 errors came from agents attempting common security exploits
- He built a public traffic-analysis site to track the crawlers
A rare first-hand dataset showing AI scraping crowding out the human web.
More from Infra
- Intel CEO Lip-Bu Tan sits down with Gavin Baker at TomorrowX Summit 2026 — BenBajarin · 2026-11-22
- Particle lands on OpenRouter with ZDR and no-China-routing privacy guarantees — AravSrinivas · 2026-11-22
- RTX 3090 FP16 unlock on Windows cuts ComfyUI image gen time by 34% — notaneimu · 2026-11-22
- Gemini Ultra's $100 plan includes 1000 monthly Colab credits—about 55 H100 hours — iskander · 2026-11-22
- Hy team pushes back on ultra-sparse MoE: sparsity recovers efficiency, not quality — douglovefishing12 · 2026-11-22
- SpaceX lenders push back on $40B of AI chips launched into space — flngr · 2026-11-22