Scrapling hits 93.6K GitHub stars as adaptive scraping toolkit for AI agents
techNmak · x · 2026-11-03
Scrapling is emerging as a serious open-source toolkit for AI agents that need to interact with the web, with 93.6K GitHub stars and 9.8K forks.
It's a single Python framework covering the entire scraping workflow — from a single HTTP request to concurrent website crawling — bundling fast HTTP requests, stealth browsers, powerful parsing, and crawling in one library.
Key highlights:
- Adaptive parsing: when a site changes its CSS classes, elements are relocated via saved properties and similarity matching, so extractors don't break;
- JavaScript rendering, persistent sessions, proxy rotation, automatic retries, pause/resume, and AutoThrottle;
- An MCP server and Agent Skill let AI agents browse and extract data directly, exporting whole sites as clean Markdown, plus CLI commands and an interactive shell.
More from coding & agent
- Grok bot calls Comcast and negotiates ~$500/year off the author's internet bill — petergyang · 2026-11-03
- Persistent Docs Make Grok Bots Remember Across Sessions, From Chat to Tesla — bfrench · 2026-11-03
- Good prompts are about articulating taste, not writing specs, argues designer — dreamwieber · 2026-11-03
- Two gaps every AI agent team must understand: requirement gap and model gap — bigdata · 2026-11-03
- Hermes Agent + MeshyAI turns conversational prompts into 3D game characters — egeberkina · 2026-11-03
- First hands-on: Meta's Muse Wonder 1.0 vs DeepSeek V4.1 Flash in Codex CLI — Richidm · 2026-11-03