How do you test a RAG support bot for wrong answers before it ships?
srinath_y · reddit · 2026-10-18
A developer crowdsources practices for testing docs-backed RAG chatbots: test set sources (hand-built, generated, real logs), judging approaches (exact match, rubrics, LLM-as-judge and how to validate it), distinguishing wrong-doc answers from hallucinations, re-running on every change, and experiences with promptfoo, DeepEval, Ragas and LangSmith.
More from coding & agent
- User says $200 OpenAI plan drained in a day, then reset oddly to zero — alexcovo_eth · 2026-10-18
- Anthropic tip: one prompt sentence changes how Claude subagents pace speed vs thoroughness — yacinelearning · 2026-10-18
- "I want an agent to minimize my diffs" — even GitHub's web UI is obsolete now — davidcrawshaw · 2026-10-18
- Dev team drops PRs entirely: push to main, review after the fact with Shelley — davidcrawshaw · 2026-10-18
- 45-year veteran coder: agentic coding is the biggest productivity jump I've seen — Scobleizer · 2026-10-18
- 80-link evidence pack shows AI agents swarming the public web: Wayback, GitHub, Wikipedia — CapitalKingGaming · 2026-10-18