Two Agents Approve, One Missing Requirement: The Trap of Who Writes Acceptance Bars
krishnan · x · 2026-11-09
Krish Subramanian examines an evaluation trap in long-running agent apps: two agents agreeing the job is done can still deliver the wrong product — a separate evaluator is useful but not automatically an independent source of truth.
Citing Anthropic's March 24 engineering post on harness design: before each sprint, a generator and evaluator negotiate a contract for what will be built and how success is checked; the evaluator then drives the running app with Playwright. The sequence is sound — the trap is who writes the acceptance bar.
Illustration: the customer wants an export of all records. The builder proposes a download button; the evaluator confirms it produces a valid file; both approve — but the export silently omits archived records. Implementation matched the contract; the contract missed the requirement. Agree on observable behavior first and test the product, not the builder's description.
More from coding & agent
- Day 21 of an agent a day: a TPP monitor agent for drug discovery, sample prompt included — nlarusstone · 2026-11-09
- Applying the bitter lesson to my LLM workflow: skip agents, just talk to the model — esrtweet · 2026-11-09
- Apple engineers' recap for local AI builders: skip AFM for coding, one-line model swapping — rxwei · 2026-11-09
- copy-clip: a copy button that folds your code into paper and clips it, built with vanilla JS — prasenx · 2026-11-09
- Dev finds sanity in talking to just one agent, one chat at a time — teodorio · 2026-11-09
- Relevant Isn't Authoritative: Why Agents Retrieve Right Docs and Still Decide Wrong — krishnan · 2026-11-09