faithgate: pytest for Prompts
ahumanbeingmars · reddit · 2026-07-05
Tired of the 'edit prompt, merge without testing' cycle, a developer created faithgate—essentially pytest for prompts: maintain a set of question/context/answer test cases, score faithfulness for a given prompt+model version, diff against baseline line by line, exit non-zero on regression, integrate with CI to block bad prompt changes at PR. Default uses Claude's built-in key as judge, metrics based on RAGAS; offline no-key mode: author publishes calibrated data (20 contradictions only 9 caught) and writes unit tests asserting that weakness. Failure means no pass: zero matches, ungraded, all wrong.
Related event: Faithgate Open-Sourced for RAG Fidelity Regression(2 posts)→
More from coding & agent
- Mirendil debuts Fern, a from-scratch model with up to 10M-token context built for AI R&D agents — shayan_ · 2026-10-27
- Firecrawl offers 2.5x credits to a16z Speedrun founders, unveils $75M Series B — devdigest · 2026-10-27
- Chums: open-source Apple Watch app puts Claude Code and Codex on your wrist — brandon_galang · 2026-10-27
- Open-source Clairic Studio turns pasted text into typed data and working code in 5 languages — dfinke · 2026-10-27
- Matt Pocock shares expand/contract trick: split one-way-door PRs to de-risk big changes — mattpocockuk · 2026-10-27
- Qdrant builds Universal I/O to run vector search mmap code on object storage — qdrant_engine · 2026-10-27