AI agents blocked by Zillow, Indeed, Reddit — a scraping API got 3/3 through in testing
eyishazyer · x · 2026-11-07
A controlled experiment: same laptop, same day, same prompt, Claude Code with default web tools failed on 3 of 5 public pages — Zillow threw a bot check, Indeed loaded one posting, Reddit errored out. "The model's fine. It just never got the pages."
Swapping in the Oxylabs Web API (via MCP) fixed all three: 9 Zillow listings, full Indeed page-1 listings, full Reddit post readable. Setup is two commands (uv tool install + npx skills add oxylabs/web-api-skills) and works with Claude Code, Cursor, Codex, or any MCP agent.
Context: Cloudflare says over half of internet traffic this year wasn't human, AI agent requests on its network grew 1,700%+ in a year, and since Sept 15 new sites block agents on ad pages by default. The author also notes Oxylabs has ISO/IEC 27001, SOC 2 Type II, GDPR support, and cyber insurance for workplace use.
Related event: AI Agents Blocked by Anti-Bot Systems on Public Websites, Tests Show(2 posts)→
More from coding & agent
- Warp built a mobile app with an AI factory in a month, cutting cost per PR from $80 to $20 — vikvang1 · 2026-11-07
- Indie dev adds Linux support via Claude Code after one premium user email complaint — yihui_indie · 2026-11-07
- SuperLedger: a single-writer decision ledger in Coda wired into Grok Bot, zero API keys — bfrench · 2026-11-07
- rexq: an open-source inbox panel that fixes Claude Code decision fatigue — nutcrook · 2026-11-07
- Stanford's Collaborative Intelligence Benchmark: Best Model Drops 20-47 Points When Teammates Misbehave — omarsar0 · 2026-11-07
- Conductor shuts down as cloud agent startups fold, devs hunt alternatives — sergeykarayev · 2026-11-07