Replacing smolagents' VisitWebpageTool with a custom fetch layer to beat anti-bot pages
InsideDebt6345 · reddit · 2026-09-10
A developer found Hugging Face smolagents' built-in VisitWebpageTool breaks on protected/JS-heavy pages — a Walmart product page returned 'Robot or human?' bot-check text that the agent then reasoned over as real content. The fix: define a custom fetchpage(url) tool with the tool decorator, document it well in the docstring (smolagents builds the tool description from it at runtime), register it in the CodeAgent tools list, and prompt the agent to use it for all retrievals. Using ZenRows (or any trusted retrieval backend) as the fetch layer gives Markdown from dynamic and protected pages without maintaining your own browser/proxy stack.
More from coding & agent
- GitHub adds enterprise-managed permissions for Copilot agent ops: shell, files, network — PaulShellDev · 2026-09-10
- Site built in 12 hours on a Mac Mini by orchestrating multiple AI agents in parallel — iannuttall · 2026-09-10
- Chess bot swap: Astra costs $20/game in API tokens, GLM-powered bot may cost pennies — MikePFrank · 2026-09-10
- Raycast v2.2 adds multi-provider AI routing, sparking debate on fair local vs hosted model evals — daani_maas · 2026-09-10
- VRAM guard trick: run a local LLM and ComfyUI video generation side by side without OOM — Sn0opY_GER · 2026-09-10
- Infocrash launches MCP server for AI-assisted structured data collection and cross-validation — polentx · 2026-09-10