Agent evals: start from observed failures, not features — a practical starter guide
ialijr · reddit · 2026-09-03
A Reddit write-up tackles the underexplained question of what to actually test in agent evals. Key points:
- The hard part is triage: which failures become eval cases, what belongs in unit/integration tests, when to use deterministic checks vs LLM judges, how many repeats, and avoiding suites that only test wording
- Framework: start from observed failures, push mechanically checkable behavior down to cheaper tests, and reserve evals for behaviors needing model-level judgment or trajectory checks
- A small suite of 5-10 good cases beats a huge imagined benchmark
The author also released a framework-agnostic guide/skill for beginners.
More from coding & agent
- fable-advisor open-sourced: Fable 5.1 architect routes GPT agents by task complexity — daniel_mac8 · 2026-09-03
- fable-advisor: Fable 5.1 as architect orchestrating GPT implementers in Claude Code — daniel_mac8 · 2026-09-03
- Every launches Compound Writing: a Claude Code plugin that compounds your editorial feedback — danshipper · 2026-09-03
- OneRegex Revera brings consistent POSIX.1-2024 ERE regex to any programming language — jedisct1 · 2026-09-03
- Every Launches Compound Writing Plugin to Make AI Writing Sessions Compound — danshipper · 2026-09-03
- video-shotcraft: open-source skill turns Claude Code into a cinematic product video studio with 157 shot recipes — tom_doerr · 2026-09-03