Agent evals: start from support-queue failures, but pair each case with a should-not twin

rahulladumor · reddit · 2026-11-05

The author reconciles conflicting advice from Anthropic, AWS and Shopify on where an in-production agent's first eval set should come from.

The split: Anthropic says 20-50 tasks from real failures (bug tracker/support queue) are a great start; AWS says build golden datasets partly from production logs, especially poor outcomes; Shopify moved away from curated golden datasets to Ground Truth Sets sampled from real merchant conversations, labelled by 3+ product experts.

Author's reading: failures answer "did we fix what broke?"; sampled conversations answer "how good is the agent on real traffic?" Start with failures, add sampling once your judge is calibrated against experts.

The one-sidedness trap: support queues are mostly "it should have done X", so the cheapest way to pass them all is over-triggering (e.g. approving refunds it shouldn't). Write a should-not twin — a near-identical request outside policy that must be refused — next to every case. As Anthropic puts it: one-sided evals create one-sided optimization.

Build checklist: order tickets by user impact; rewrite each as one unambiguous task with a reference solution (0% pass rate over many trials means a broken task); grade in code where rules fit; validate any LLM judge against human labels and audit for drift; graduate stable cases into a regression suite.

Redaction gotcha: CloudWatch Logs data protection is a masking view over stored logs — pre-policy ingested events aren't masked, some identifiers need nearby keywords, and free-text business detail has no managed category. Redact when cases enter the repo.

Original post →

Agent evals: start from support-queue failures, but pair each case with a should-not twin

More from coding & agent