谷歌 Gary Illyes 谈假 Googlebot 爬虫泛滥与 AI 训练抓取之乱
gaganghotra_ · x · 2026-10-29
Google's Gary Illyes joined Aneta Samkov on the Odys Podcast to break down crawler impersonation and verification: a Googlebot User-Agent proves nothing (spoofing it is "literally one line of code"); real verification requires official IP ranges and reverse DNS; he uses honeypot sites to catch fake Googlebots; fake crawlers do things the real Googlebot never would; AI companies' hunger for training tokens has made bad crawling worse; unknown crawlers can exhaust bandwidth and take sites down. He also covered who pays for bot authentication (WAFs, CDNs, or site owners) and described the internet as "a farm to harvest."
「Infra」频道最新
- 单张 RTX 5080 跑 Qwen3.8 Flash Next：IQ3_S 量化峰值 275 tok/s — yehyakar · 2026-10-29
- 谷歌云 CEO：苹果在最新 TPU 上扩展基础模型预训练 — shayan_ · 2026-10-29
- 一张图看懂 FlashAttention:瓶颈不是计算而是搬运数据 — cneuralnetwork · 2026-10-29
- Particle-1 推理服务上线 OpenRouter,输出 token 省 28-52% — ycombinator · 2026-10-29
- Qwen3.8-27B EXL3 量化 KLD 仅 0.0002,18.8GB 显存超 Q8 精度 — suprjami · 2026-10-29
- MLX-Serve 全面重写:插件架构支持任意模型与量化格式免重编译接入 — TheMoonMidas · 2026-10-29