Meta crawler hammers one site with 700k requests/day; Cloudflare adds x402 pay-per-crawl
kleffew94 · x · 2026-10-06
- Developer @joshtriedcoding reports Meta's crawler spamming Upstash with 700,000+ requests in a single day, many fetching the same pages hundreds of times.
- @rohinlohe argues the (abusive) crawler should foot the compute/bandwidth bill: charge crawlers by User-Agent or block them outright.
- Context: Cloudflare's Monetization Gateway applies x402 payment requirements to matching traffic, protecting APIs, MCP tools, sites, and datasets — buyers sign in-request cryptographic payment authorizations settled via Coinbase's x402 Facilitator, with rules that can target verified bots only.
More from Infra
- Relace AI Reportedly Served 2.3T Tokens on OpenRouter in a Day, Topping OpenAI's 1.8T — stuffyokodraws · 2026-10-06
- Cloudflare adds natural-language observability charts, plus 9 ready-to-use debugging prompts — hichaelmart · 2026-10-06
- Cloudflare's X402 protocol makes AI agents pay per scrape, feeding an agentic economy on stablecoins — kleffew94 · 2026-10-06
- YuE2 music model runs fully in-browser: 40s song in ~60s on a MacBook via WebGPU — realmrfakename · 2026-10-06
- DeepSeek-style agent infrastructure ran 1.3B sandboxes in 4 weeks, peaking at 170K concurrent — teortaxesTex · 2026-10-06
- Perplexity CEO: power users now burn $10k+/month of compute in agent loops — rohanpaul_ai · 2026-10-06