RAG Went From 4/10 to 9/10 Correct — Testing Client Data First Revealed It Was Never a Model Problem

max_gladysh · reddit · 2026-09-22

Before building a RAG system for a fifth-generation textiles client (10,000+ products, 1,000+ internal docs, live inventory), the BotsCrew team piloted with real employee questions. Only 4 in 10 answers were correct — none due to the model:

Conflicting documents with no way to tell which version is current
Unreadable scans indexed alongside good content
Retrieval returning the plausible document instead of the authoritative one
Business rules existing only in employees' heads

Fixes: split one retrieval path into routing — product questions to exact-match catalog search, stock to a live inventory API, policy/general to a cleaned RAG layer ("a lot of RAG accuracy problems are routing problems in costume"). Mapping internal acronyms and region codes gave the single biggest jump. A fixed test set made it measurable: from under 60% to roughly 90%.

A later wrinkle: the warehouse backfills, so the same question a week apart returned different numbers — the team had to agree on what "correct" even meant. The takeaway: AI transformation is mostly about whether your data is where you think it is, whether processes exist outside people's heads, and who decides when documents disagree. A pilot is the cheapest readiness assessment you'll ever run.

Original post →

RAG Went From 4/10 to 9/10 Correct — Testing Client Data First Revealed It Was Never a Model Problem

More from coding & agent