Google's Gary Illyes on fake Googlebot crawlers, honeypots, and the AI scraping boom
gaganghotra_ · x · 2026-10-29
Google's Gary Illyes joined Aneta Samkov on the Odys Podcast to break down crawler impersonation and verification: a Googlebot User-Agent proves nothing (spoofing it is "literally one line of code"); real verification requires official IP ranges and reverse DNS; he uses honeypot sites to catch fake Googlebots; fake crawlers do things the real Googlebot never would; AI companies' hunger for training tokens has made bad crawling worse; unknown crawlers can exhaust bandwidth and take sites down. He also covered who pays for bot authentication (WAFs, CDNs, or site owners) and described the internet as "a farm to harvest."
More from Infra
- Qwen3.8 Flash Next at IQ3_S hits 275 tok/s on a single RTX 5080 via Basalt fork — yehyakar · 2026-10-29
- Google Cloud CEO: Apple Scaling Foundation Model Pretraining on Latest TPUs — shayan_ · 2026-10-29
- FlashAttention vs standard attention in one chart: the bottleneck is data movement, not compute — cneuralnetwork · 2026-10-29
- Particle-1 inference goes live on OpenRouter, claiming 28-52% fewer output tokens — ycombinator · 2026-10-29
- Qwen3.8-27B EXL3 quant hits lowest KLD of 0.0002 in just 23 GB — suprjami · 2026-10-29
- MLX-Serve 26.11.1 rewrite: plugin architecture adds models and quants without recompiling — TheMoonMidas · 2026-10-29