kernel.org burns 14 CPU cores just rendering pages for abusive AI crawlers
Simon Willison · rss · 2026-09-08
Konstantin Ryabitsev of kernel.org (the official Linux kernel Git repository) details how bad the "background radiation" of abusive crawlers has become:
- CPU cycles spent rendering commit pages for scrapers now exceed all legitimate traffic combined, including git clones
- At any moment, across 5 geo-distributed nodes, 14 CPU cores do nothing but render git commits as HTML for crawlers
Simon Willison notes the same concern applies to Datasette, which serves many crawlable pages. AI-training-driven crawler floods are becoming a real burden on small open source infrastructure.
More from Infra
- AMD's Halo Station Touts 2.6TB Memory, Tripling NVIDIA's 748GB DGX Station, Shipping 2027 — shashib · 2026-09-08
- DavidSHolz: AI services should let users pay extra to spin up CPU/GPU VMs — DavidSHolz · 2026-09-08
- What broke running an agent fleet 24/7 — it wasn't prompt quality — ParrotIntegrated · 2026-09-08
- Dual RTX 3060 vs single RTX 3090: choosing a $500 vs $1000 build for a home local LLM server — Bakkario · 2026-09-08
- Huawei 950DT reportedly priced at ~$16K per chip in DeepSeek order, below expectations — teortaxesTex · 2026-09-08
- Report: Intel to raise PC CPU prices 10%, may cut another 5-10% of workforce — zephyr_z9 · 2026-09-08