Dev agency shares a rigorous eval framework for agent harnesses built on Claude Code

DevFizzah · reddit · 2026-11-17

A dev agency building an internal harness around Claude Code (structured spec in, deployed web app out) asks how to objectively compare harness versions with the model fixed. Their plan: frozen golden specs, human-written holdout e2e/API tests measuring "false-ready" rate, changeability tests via a fixed baseline agent applying change requests, blind pairwise LLM review, and ≥3 runs per config reporting median and pass^k. They also pose open questions on holdout leakage, code-quality metrics, judge calibration, and run variance.

Original post →

More from coding & agent

coding & agent channel →