faithgate: Regression Testing Gate for RAG Faithfulness
ahumanbeingmars · reddit · 2026-07-06
After a prompt change silently caused their RAG bot to hallucinate without errors for days, the author open-sourced faithgate. It acts as a regression gate for faithfulness: maintaining a set of question/context/answer test cases to score every prompt or model version. If grounding drops below the baseline, CI fails. It uses the RAGAS Faithfulness metric under the hood with Claude as the default judge. Testing on a 20-document corpus with three planted hallucination types (date swaps, entity swaps, and baseless splicing) caught them all, dropping scores from 1.00 to 0.29, 0.12, and 0.20. The author notes the offline keyless judge mode is weaker, catching only 9/20 unfaithful answers in 40 manual labels, and includes unit tests asserting this number.
Related event: Faithgate Open-Sourced for RAG Fidelity Regression(2 posts)→
More from coding & agent
- LangChain and Tavily to co-host NYC event on production agents using live web data, Nov 12 — LangChain · 2026-10-26
- Dev hacks reMarkable e-ink tablet to let Opus draw and animate — Stefania_druga · 2026-10-26
- Godot games are trivially decompilable — compile C# to native DLL for real protection — mflux · 2026-10-26
- Techhalla shares full 2D-to-3D workflow with Opus 5.5 and Kling 4.0 prompts — techhalla · 2026-10-26
- Jason Furman: Building Econvitals pre-AI would need ~50 people, 3 years, $20M — HarrySurden · 2026-10-26
- Dev dictates to Claude Code for 4+ hours while riding indoor bike in aero position — jacob_posel · 2026-10-26