Dumping 3,000 pages into your agent's vector store makes retrieval worse — filter hard first
Desperate-Talk-4398 · reddit · 2026-10-10
The author recounts watching someone crawl an entire domain (3,000 pages) into a vector store and get worse retrieval than with 40 pages — because most of a website is nav, legal text, and stale blog posts, and unfiltered crawlers just pour noise into your index.
What works, per the author: crawl only to discover URLs, filter hard on path before fetching any page, then pull content only for survivors. His filter is deliberately simple — one prefix and two exclusions cut 3,000 pages down to 250 that matter.
He argues "index everything and let retrieval sort it out" is right for search but wrong for agents, which have a task, a token budget, and no patience.
More from coding & agent
- Ethan Mollick's new book charts 30 months of AI progress, from cyborg to conjurer work — rseroter · 2026-10-23
- OpenJet claims Qwen3.8-27B coding runs 1.9x faster than llama.cpp on one RTX 3090 — JeremyCMorgan · 2026-10-23
- Jensen Huang details Vera Rubin agentic stack: NVL72, MGX, CMX and RTX — rohanpaul_ai · 2026-10-23
- Building an autonomous SEO agent in Hermes with three skills: programmatic pages, Keywords Everywhere, DataForSEO — EXM7777 · 2026-10-23
- Building a sports betting AI agent with onchain execution and human approval by default: architecture and lessons — Dry-Comfortable-2678 · 2026-10-23
- Capy 0.5.0 lets Claude Pro/Max subscribers reuse their plan quota, adds OIDC cloud access — garrytan · 2026-10-23