Dev agency shares a rigorous eval framework for agent harnesses built on Claude Code
DevFizzah · reddit · 2026-11-17
A dev agency building an internal harness around Claude Code (structured spec in, deployed web app out) asks how to objectively compare harness versions with the model fixed. Their plan: frozen golden specs, human-written holdout e2e/API tests measuring "false-ready" rate, changeability tests via a fixed baseline agent applying change requests, blind pairwise LLM review, and ≥3 runs per config reporting median and pass^k. They also pose open questions on holdout leakage, code-quality metrics, judge calibration, and run variance.
More from coding & agent
- Learn system design by building a URL shortener with AI and interrogating its limits — cneuralnetwork · 2026-11-17
- Clare Liguori to lead AWS reInvent Developer keynote on Nov 30 — FlolightC · 2026-11-17
- VS Code retires its Local agent harness in February 2027, replaced by Copilot harness — Formal-Grass-3173 · 2026-11-17
- Courio MCP server lets AI agents print and mail physical letters worldwide — benoittallandier · 2026-11-17
- REA-Video skill lets Claude reverse-engineer video editing styles; rival claims 20x speed at SSIM 0.96 — toolstelegraph · 2026-11-17
- Developer retries Amp Code after months and finally gets the hype: "completely different" — g3off_who · 2026-11-17