harness-evals: open-source library to compare agent versions with success-vs-cost plots

slavadubrov · reddit · 2026-11-16

harness-evals answers one question: after changing your agent (a prompt, a tool, an approval step), did it get better? You write tasks with checks plus a small adapter running your agent; the library runs each version multiple times and reports success rate, pass^k, cost and latency with intervals, a success-vs-cost plot, and a failure map. Zero dependencies, MIT, with a companion blog post walking through the author's usage.

Original post →

More from coding & agent

coding & agent channel →