Running 1000s of enterprise RAG indexes: a year of lessons in incremental sync
srnsnemil · reddit · 2026-09-29
A founder of kapa.ai (YC S23), which hosts thousands of AI knowledge indexes over docs, Slack, tickets and wikis, shares a year of hard-won lessons after replacing nightly re-indexing with incremental sync across 20+ source types:
- Every source defines "changed" differently: sources with delta APIs (Zendesk, Confluence) sync every 5-10 min; Slack every 6h via 7-day rescans; web crawls daily or weekly.
- Web crawling pitfalls: build numbers, footer years and nav versions change constantly while content doesn't — reprocessing on HTML diff means a hidden daily full reindex. The fix: compare extracted content, not raw HTML.
- Metadata lies: lastmodified timestamps churn on unchanged files, wasting processing; compare actual content where possible.
- Deletions are the hard part: almost no API answers "what was deleted?", so full listing comparisons run slowly (24-48h for most sources), during which deleted pages may still be cited.
- Big changes go to a human first: a site redesign returning 404s makes the crawler conclude "everything deleted" and can wipe the index — if ≥45% of pages change or vanish in one crawl, nothing is applied until reviewed.
A candid engineering retrospective with concrete numbers, highly relevant for teams building RAG sync pipelines.
More from coding & agent
- Glasser launches pay-per-call API marketplace: ~2000 paid endpoints under one key — testingcatalog · 2026-09-29
- Dev builds content strategy agent with Hindsight memory layer that learns from past content performance — vikramsaiandra · 2026-09-29
- Agent spins up 322 Hugging Face Jobs in 90 minutes for about $4 — victormustar · 2026-09-29
- Dev builds local Windows voice assistant with 133 tools and hard risk gates the LLM can't bypass — Safe-Cucumber-9316 · 2026-09-29
- Codex TUI's single-daemon switch slammed: permissions reset, no multi-credential sessions — moyix · 2026-09-29
- 7 Essential Claude Connectors to Stop Copy-Pasting Between Your Apps — alifcoder · 2026-09-29