AI Crawlers Hit Site With 40,000 Requests a Day; Owner Pays 10x Infra Costs
burkov · x · 2026-10-07
ML author Andriy Burkov says ChapterPal serves 120,000 requests daily but only 11,000 come from humans — ClaudeBot and PerplexityBot together make 40,000 requests a day, forcing him to pay roughly 10x his real infra cost. He blocked all bots except Googlebot, Bingbot, and social-media metadata fetchers, reigniting the debate over AI crawler compensation and robots.txt etiquette.
More from Infra
- Serving & Monitoring Local LLMs Across Three Mixed-GPU Machines Without Duct Tape — ziyaulhuk12 · 2026-10-07
- Disaggregated inference is the future, says e/acc's Beff Jezos after panel with General Compute — beffjezos · 2026-10-07
- What Does 'Owning Your Own AI' Technically Mean? Locally Running Open Weights Debated — Imaginary_Choice_430 · 2026-10-07
- NVIDIA's LoGRA cuts LLM RL training memory by up to 45.7%, enables 27B RL on a single 8-GPU node — nvidia · 2026-10-07
- Ex-NVIDIA engineer tells the story of testing a chip with a broken memory controller — blelbach · 2026-10-07
- Follow the money risk: memory suppliers take deposits, AI chip makers finance their customers — tengyanAI · 2026-10-07