Open-source scraper escalates from plain HTTP to browser TLS to real browser, with MCP server
vhsgreed · reddit · 2026-11-01
The author open-sourced certainlyscraped, a lazily-escalating scraper: it tries plain HTTP first, retries with a browser TLS fingerprint if refused, and only spins up a real browser as a last resort. Each page records which escalation step was needed, so failures are diagnosable instead of silent.
It also runs as an MCP server so agents can call it directly. A nightly table of 39 popular sites tracks which ones return data over plain HTTP vs. which require JavaScript — most major social platforms need browser-level fetching.
More from coding & agent
- Manus growth lead Will: agents will outnumber people on growth teams — gefei55 · 2026-11-01
- mitsuhiko shares screenshot of AI agents politely apologizing to each other mid-work — mitsuhiko · 2026-11-01
- Dev: most "AI tools for reverse engineering" are slop — agents prefer objdump plus sqlite — mgostIH · 2026-11-01
- Dev rants: ChatGPT still won't let you create persistent skill files it can access — JeremyNguyenPhD · 2026-11-01
- AI-built Star Wars parody game SLOP EISLEY ships 0.6.70 update — jasonkneen · 2026-11-01
- Open-source memory layer ICE loses to vector-RAG on LongMemEval: 43% vs 69.5% — Deepnar_S · 2026-11-01