Dev open-sources a dependency-free LLM eval harness that catches regressions, not averages

zahidsaim23 · reddit · 2026-09-07

A developer who evaluates LLM output daily has open-sourced a tiny, dependency-free eval harness (MIT): github.com/zahid23saim/llm-eval-harness.

How it works: your eval is a JSON gold set — each item has a question, the correct answer, and a match rule (exact / contains / numeric). Run it on every prompt or model change; it prints accuracy, exactly which items failed and why, and exits non-zero to fit into CI. Standard library only.
Lesson 1: the number that matters after a change isn't the average score — it's which items used to pass and now fail. A change can raise the average while breaking the three questions your biggest user depends on, so diff runs and watch for regressions.
Lesson 2: for open-ended answers, string matching gives up fast. An explicit rubric graded by a model at temperature 0 is far more stable than "is this good?" — but you still need to spot-check the judge against a handful of human labels, or it will confidently lie.

More from coding & agent