Meta's crawler hit 15x Googlebot volume in internet-scale incident, now fixed via Vercel
cramforce · x · 2026-10-15
Developer cramforce (Vercel) revealed an internet-scale incident: Meta's crawler (meta-externalagent) sent excessive, often repetitive requests at over 15 times Googlebot's baseline volume. The chart shows hourly requests from Sep 27 to Oct 14, with meta-externalagent far above googlebot's flat baseline and peaking around Oct 3-5.
Vercel worked with the Meta team to normalize the crawling volume, and the issue is now resolved. A notable example of AI training crawlers straining web infrastructure.
More from Infra
- Mine Bittensor Subnet 92 by pointing Claude Code at one link — airesearch12 · 2026-10-15
- OpenAI infra lead leaves to start up after locking in double-digit GWs of compute — DesireeCachette · 2026-10-15
- Who Sells Data directory maps 159 AI training data vendors, compares Scale vs Mercor — cneuralnetwork · 2026-10-15
- Compute is the next commodity: building financial infrastructure for GPU markets — AnneliesGamble · 2026-10-15
- OCP slide reveals NVIDIA Rubin Ultra NVL576 will use CPO for scale-up networking — BenBajarin · 2026-10-15
- TensorFold ships macOS app for local LLM inference, reports 874 tok/s on RTX PRO 6000 — HankYeomans · 2026-10-15