Test: AI agent blocked on 3 of 5 public sites until routed via scraping API
eyishazyer · x · 2026-11-07
eyishazyer ran a controlled test exposing a hidden AI agent weakness: pages you can open in a browser may be blocked for your agent. Same laptop, same day, same prompt, 4 public pages (Zillow, Indeed, Reddit, Python docs): Agent A — Claude Code with default web tools, no browser — failed to open 3 of 5 sites; Agent B — the same agent wired to Oxylabs Web API — fetched them fine. Takeaway: default web tools get caught by anti-bot walls, so agents need a scraping API for reliable access. Note the post doubles as Oxylabs promotion, so verify tool choice independently.
Related event: AI Agents Blocked by Anti-Bot Systems on Public Websites, Tests Show(2 posts)→
More from coding & agent
- Anthropic rolls out conversational Claude Code Projects with waitlist for paid users — lydiahallie · 2026-11-07
- Claude Code Projects can replace Conductor workflows, minus multiplayer — lydiahallie · 2026-11-07
- Anthropic details how Claude "dreams" to organize its memory — round · 2026-11-07
- Seroter Daily: Agent Collisions Beyond Worktrees, 2026 Enterprise AI State, Cursor Deployments — rseroter · 2026-11-07
- After 48 hours with Amp Code, this user says it beats Conductor — HankYeomans · 2026-11-07
- Founder builds a Claude Code "PR factory" that pitches journalists automatically — doooyle · 2026-11-07