Redditor shares 16-ticket injected-bug eval harness for coding agents, asks if it's overkill
Infinite-Fly-2756 · reddit · 2026-11-03
A developer details a deterministic eval setup for custom QA/coding agents: a fixed corpus of 16 coding tickets with reference implementations, injected bugs verified by hidden tests, and metrics like fail-for-the-right-reason rate, bug-catching accuracy, rule adherence and pass^k/token cost across trials with 95% CIs.
The pain: every prompt tweak or model swap (Sonnet/Haiku) requires rerunning the full corpus, which is compute-heavy. They ask production users whether a custom harness is necessary, whether LLM-as-a-judge works for coding agents, and for lightweight framework recommendations like Promptfoo or DeepEval.
More from coding & agent
- Ant Group's SWE-Trail: history-guided trajectory collection yields better coding agents — antgroup · 2026-11-03
- Open-source AI Xianyu monitor with Playwright + LLMs hits 16k GitHub stars — Usagi-org · 2026-11-03
- Researchers Wrote a Paper Assessing Matt Pocock's /retro Skill for Coding Agents — mattpocockuk · 2026-11-03
- Developer kicks off a Worms remake built with AI coding tools — ilumine_ai · 2026-11-03
- Creator hires an AI employee 'Viktor' in Slack to fix Claude's teamwork gap — rubenhassid · 2026-11-03
- Surrogate Inference Engine Accused of License-Washing Open-Source Project NInfer — teortaxesTex · 2026-11-03