Getting good at evals: Build a taxonomy of failure modes
realmadhuguru · x · 2026-08-20
Part 3 of a series on getting good at evals. The author advises that once you have v1 of your evals, the first thing to do is build a taxonomy of your AI system's failure modes. The process involves studying the last 500-1000 production traces, focusing on failures, and clustering/naming them. The article emphasizes specificity: instead of "bad answer," failures should be categorized precisely, such as "wrong document retrieved," "right document but irrelevant section," "failed to ground to context and hallucinated," "failed to punt and instead made stuff up," or "ambiguous question with poor assumptions." Precise naming allows for designing eval tests to catch specific failures, bridging the gap to an improvement flywheel.
Related event: Eval Guide: Build a Failure Taxonomy to Drive AI Improvements(2 posts)→
More from coding & agent
- Cursor adds /cloud and /local commands for environment switching — mattyp · 2026-08-20
- Metabigor: Infrastructure Mapping Tool Without API Keys — tom_doerr · 2026-08-20
- HITL Persists in Agents Solely Due to Lack of Evals — verrsane · 2026-08-20
- Dev argues AI code is unreliable, refutes compiler analogy — blelbach · 2026-08-20
- Claude Code adds Concise output style: result first, shorter responses — EricBuess · 2026-08-20
- Coding agents let senior researchers do first-author work again — anirudhg9119 · 2026-08-20