100 Web Scraping Tools Curated for AI Work, Plus Oxylabs' Open-Source Agent Skills
Aiden_Tech_Ai · x · 2026-10-31
A curated list of 100 web scraping and data collection tools spans 10 categories: proxy platforms (Oxylabs, Bright Data), scraping APIs (ScraperAPI, ZenRows), no-code tools (Octoparse), crawler frameworks (Firecrawl, Crawl4AI, Scrapy), SERP data (SerpApi, Tavily), e-commerce data, workflow automation (n8n), and storage (DuckDB, Pinecone).
Alongside it, Oxylabs open-sourced web-api-skills (MIT), providing skills that teach coding agents to use its Web API for live search and fetching, a structured research method (search, pick sources, read, cite), and a migration guide from Tavily/Exa/Firecrawl/Perplexity/Brave/Linkup. Installable via npx skills add for Claude Code, Cursor, Codex and other skills-aware agents.
More from coding & agent
- Claude iterated on its own screenshots against 341 real photos to build a browser island — prasenx · 2026-10-31
- Filmmaker used OpenClaw to organize notes and win a grant: workflow beats tools — bytebot · 2026-10-31
- NVIDIA Extends Agent Toolkit to Engineering: PhysicsNeMo Goes Agent-Ready — emmanuelvivier · 2026-10-31
- The internet is just search and action—and agents will make apps obsolete — yunta_tsai · 2026-10-31
- Memnest: open-source local MCP memory server shared by Claude Code, Codex and Cursor — Agitated_Chair_4977 · 2026-10-31
- Attacker Used BYOK Agent Strix to Hit China/HK Gambling Sites, Leaked Own LLM Key — cyb3rops · 2026-10-31