faithgate: pytest for Prompts
ahumanbeingmars · reddit · 2026-07-05
Tired of the 'edit prompt, merge without testing' cycle, a developer created faithgate—essentially pytest for prompts: maintain a set of question/context/answer test cases, score faithfulness for a given prompt+model version, diff against baseline line by line, exit non-zero on regression, integrate with CI to block bad prompt changes at PR. Default uses Claude's built-in key as judge, metrics based on RAGAS; offline no-key mode: author publishes calibrated data (20 contradictions only 9 caught) and writes unit tests asserting that weakness. Failure means no pass: zero matches, ungraded, all wrong.
Related event: Faithgate Open-Sourced for RAG Fidelity Regression(2 posts)→
More from coding & agent
- Microsoft’s ReOPD reuses teacher prefixes to make agent distillation 4× faster — dair_ai · 2026-07-27
- One developer merges Claude Code and Codex histories into a 10-million-message archive — doodlestein · 2026-07-27
- Google roundup lists 15 free AI tools across marketing, coding, docs and music — aigclink · 2026-07-27
- A 9B Ollama agent can run a fully local DJ radio with tools, memory, and TTS — pinku1 · 2026-07-27
- Bugbot rejects an MCP permission flag because it would break path-scoped isolation — zeeg · 2026-07-27
- One GPT-5.6 agent is guarding a Blink security system while another makes a parody rap album — repligate · 2026-07-27