An async agent for search and scraping needs a better tool architecture
flock-of-nazguls · reddit · 2026-07-28
Building an async agent for search, scraping, and source ranking
The author wants to automate fact-finding by having an agent search the web, follow leads, fetch referenced pages, extract answers, and rank sources by reputation.
Key points from the post:
- the problem is often solvable with custom scripts, but some facts are not indexed;
- Claude worked on open sources but refused to crawl pages blocked by robots rules;
- the author is experimenting with local Qwen models on a Strix Halo setup via llama-swap;
- they are unsure whether to build a custom agent loop with tools and a sandbox, or rely on existing bulk scraping/crawling tools;
- the system is fully async and queue-driven, so accuracy matters more than speed.
The real question is architectural: should the agent curate its own scraping tools on demand, or should the pipeline start from traditional crawl/scrape infrastructure and then extract results from the file tree?
More from coding & agent
- Netlify Says 60% of New Users Deploy Their First Site Through Drop — thisiskp_ · 2026-07-28
- Coding agents are already reshaping empirical social science research — soumitrashukla9 · 2026-07-28
- A coding-agent workspace shows coordinators, verifiers, and task specs — Wattenberger · 2026-07-28
- Netlify says 60% of June 2026 signups made their first deploy through Drop — thisiskp_ · 2026-07-28
- Claude as PM, Codex and Gemini as workers, with done enforced in code — Seunghyeon413 · 2026-07-28
- Emblem says it can build an 80-slide cited deck in 35 minutes instead of 100 hours — rohanpaul_ai · 2026-07-28