Indie dev ships local RAG desktop app: 5 hard-won lessons on citations and hallucination checks

FreakFrakFrok · reddit · 2026-11-07

The developer shipped ScrapeVantageAI Desktop (now on the Microsoft Store), a Windows app that answers questions over local PDFs, docs and slides, and shared five engineering lessons:

Index locally, send only evidence: files are parsed and embedded on-device; only relevant snippets (tagged with file name and page) leave the machine, keeping the privacy story simple and forcing page-level citations.
A citation marker is not verification: the first version treated [n] chips as verified, but models cite loosely and polite filler counted as verified claims. Fix: judge each claim against its own cited fragment and don't count filler as claims.
"No invention" detectors over-fire on framing: on 85 real outputs it produced 6 flags, all false positives; excluding framing sentences brought it to 1 with injected fabrications still caught at the same rate.
Regex can't tell an honest gap from an invention: a small LLM judge that only reviews flagged sentences and can only remove flags approved 10/12 honest gaps and 0/12 disguised inventions, at 0.4s per call.
Local jobs must never block the user: indexing state and chat now run independently so you can query whatever is already indexed.

Original post →

Indie dev ships local RAG desktop app: 5 hard-won lessons on citations and hallucination checks

More from coding & agent