Agent evals have two axes, not one: graders are the layers, repeated trials cut across all of them

bharadwajp · reddit · 2026-11-23

A former SDET now working on LLM apps argues the usual testing pyramid folds two axes — how much of the system runs and who gives the verdict — into one stack. Graders are the layers: code checks at the base, calibrated model judges in the middle, people at the top. Repeated trials sit behind every layer, because a ledger check on a stochastic agent needs repetition as much as a judge does.

The running example is a refund agent whose reply said "refunded" while the ledger showed nothing — a code check caught it; a judge reading only the reply would have passed it. Full write-up at bharad.dev/blog/eval-pyramid.

Original post →

More from coding & agent

coding & agent channel →