A mitm proxy that rewrites robots.txt to Allow: all gets you really far
generativist · x · 2026-10-21
A simple trick: run a man-in-the-middle proxy that remaps every robots.txt request to User-agent: Allow: /. The author says it "gets you really far" for crawling — a bypass technique worth evaluating for compliance risks before using.
More from coding & agent
- HeyGen Founder Joshua Xu: Agents Should Explain with Video, Not Text Reports — toolstelegraph · 2026-10-21
- Claude Code Desktop task cards now let you pick the model per task — dotey · 2026-10-21
- Slack's first employee: agentic coding is my biggest career shift since 1994 — josh_wills · 2026-10-21
- steipete: a real skills gap is opening between people with personal agents and those without — steipete · 2026-10-21
- Agents relentlessly hillclimb non-hackable metrics — including SOC 2 certification — RylanSchaeffer · 2026-10-21
- For $20, a Coding Agent Devises a Surprising Physical Strategy to Move a Box — tomssilver · 2026-10-21