Building Great Evals Part 10: Measure the Steps, Not Just the Result
realmadhuguru · x · 2026-09-10
- Core claim: agent evals must measure trajectories, not just final answers. Two runs may both yield 42, but one uses 4 clean tool calls and correct retrieval while the other repeats searches 3 times, recovers from 2 errors, and burns 17 calls.
- How-to: (1) define the whole workflow; (2) define tasks per step; (3) decide how to measure each step — standalone evals or slices of a bigger eval; (4) define median and hard tasks and reflect them in evals.
- Part 9 recap (the Eval Roadmap problem): most evals fail because teams treat them as static artifacts while user expectations evolve. A financial research agent goes from summarizing a 5-page earnings report → comparing 5 reports → building an investment thesis from 15 filings → monitoring a portfolio with alerts; evals need a roadmap that evolves with the product.
More from coding & agent
- Okibi Launches on YC: Turns Your Product Into a CLI for AI Agents — ycombinator · 2026-09-11
- Claude Code desktop app lets you pop out diff and terminal panes into their own windows — ClaudeDevs · 2026-09-11
- DHH proposes video-game badges for sub-minute build times — AnushElangovan · 2026-09-11
- Redditor Wants LLMs to Debate Each Other Instead of Manual Copy-Paste — crua9 · 2026-09-11
- GitHub Copilot Day live streams showcase Copilot App and VS Code updates — martinwoodward · 2026-09-11
- Cerebras Fast Inference Flips Agent Workflows: Fewer Parallel Agents, Same Output — MatthewBerman · 2026-09-11