faithgate: Regression Testing Gate for RAG Faithfulness
ahumanbeingmars · reddit · 2026-07-06
After a prompt change silently caused their RAG bot to hallucinate without errors for days, the author open-sourced faithgate. It acts as a regression gate for faithfulness: maintaining a set of question/context/answer test cases to score every prompt or model version. If grounding drops below the baseline, CI fails. It uses the RAGAS Faithfulness metric under the hood with Claude as the default judge. Testing on a 20-document corpus with three planted hallucination types (date swaps, entity swaps, and baseless splicing) caught them all, dropping scores from 1.00 to 0.29, 0.12, and 0.20. The author notes the offline keyless judge mode is weaker, catching only 9/20 unfaithful answers in 40 manual labels, and includes unit tests asserting this number.
Related event: Faithgate Open-Sourced for RAG Fidelity Regression(2 posts)→
More from coding & agent
- Bugbot rejects an MCP permission flag because it would break path-scoped isolation — zeeg · 2026-07-27
- One GPT-5.6 agent is guarding a Blink security system while another makes a parody rap album — repligate · 2026-07-27
- An agent got unblocked by reusing a logged-in browser, not stealth tricks — armanidev_ · 2026-07-27
- Paper argues graph topology can become the core operating system for AI agents — theomitsa · 2026-07-27
- Claude Code desktop adds UI markup feedback for smoother visual editing — EricBuess · 2026-07-27
- Anthropic says Claude Code can drop 80% of its system prompt with no coding loss — krishnan · 2026-07-27