A/B your agent harness: score accepted results per dollar on hidden cases to know if improvements earn their cost
krishnan · x · 2026-11-11
The author proposes a concrete pilot: run fixed vs editable agent harness on the same workflow and total budget, freeze both, score new hidden cases, and count accepted results per dollar including improvement runs. Re-run old cases to catch regressions. The core claim: a rewritten prompt is cheap—knowing whether to keep it is the product. If gains vanish on unseen work, the loop merely learned the exercise.
More from coding & agent
- Anthropic's real bet is MCP connectors embedding HubSpot, Adobe and Atlassian inside Claude — petrusenko_max · 2026-11-11
- ZenML Pro launches always-on AI engineering team to keep agents in production — strickvl · 2026-11-11
- Latent Space tests NaiveAI's 2,100 tok/s coding model: fast, verbose, promising — Latent Space · 2026-11-11
- Anaconda's LLMOps guide: retrieval quality degrades independently of the model — anacondainc · 2026-11-11
- Dev demos agent with 'durable object brain' plus on-demand computer, local or remote — samgoodwin89 · 2026-11-11
- Right-sizing intelligence per role: frontier reasoning at the control plane, adversarial verifiers at the gate — clmt · 2026-11-11