A local LLM eval harness for customer support reveals how prompts regress
Product_Enthusiast24 · reddit · 2026-07-25
The author built a local LLM eval harness as a hands-on way to learn how evaluation systems work.
What the project tested
- A customer-support scenario built from dummy data and mock service cases
- Evaluation targets such as policy adherence, answer accuracy, grounding, and regression handling
- Hallucination checks using dummy documentation with intentionally missing answers
How it was set up
- Local model execution through tools like Ollama and local API endpoints
- Repeated, cost-free test runs during prompt iteration
- Streamlit UI for metrics, run history, and side-by-side failure inspection instead of terminal-only JSON logs
What the author learned
When a system prompt was changed to fix one edge case, it often introduced regressions in cases that had previously passed. The post closes by asking how such a local setup would differ in production.
More from coding & agent
- OpenAI Blocks AI Agent After 33-Hour Attempt to Prove Fermat's Last Theorem — MikePFrank · 2026-07-25
- Vjeux says agent session reviews help, but fully autonomous fix loops still break down — Vjeux · 2026-07-25
- How to keep local AI evals useful when models keep changing — Sufficient-Curve4753 · 2026-07-25
- 10 agent eval patterns every AI engineer should know, from golden sets to trajectory scoring — Roger_M_Taylor · 2026-07-25
- Visa open-sources a cybersecurity harness that can plug into any model — Roger_M_Taylor · 2026-07-25
- PixelRAG skips HTML parsing, uses screenshots for web retrieval, and beats text RAG by 18.1% — Roger_M_Taylor · 2026-07-25