Agent evals have two axes, not one: graders are the layers, repeated trials cut across all of them
bharadwajp · reddit · 2026-11-23
A former SDET now working on LLM apps argues the usual testing pyramid folds two axes — how much of the system runs and who gives the verdict — into one stack. Graders are the layers: code checks at the base, calibrated model judges in the middle, people at the top. Repeated trials sit behind every layer, because a ledger check on a stochastic agent needs repetition as much as a judge does.
The running example is a refund agent whose reply said "refunded" while the ledger showed nothing — a code check caught it; a judge reading only the reply would have passed it. Full write-up at bharad.dev/blog/eval-pyramid.
More from coding & agent
- wllama v4.0 ditches wasm mem64 for mem32 (up to 22% faster), 2.8x speedup on Chrome — ngxson · 2026-11-23
- Replit agents speed up NumPy 18.9x — but only Claude's team scales — amasad · 2026-11-23
- A better yardstick for 10x engineers: how much of their work gets reused — vboykis · 2026-11-23
- Microsoft rebrands Azure AI Search as Foundry IQ, a knowledge layer for agents — adnan_hashmi · 2026-11-23
- Gemini agent builds rigged Optimus robot in Blender with 145 parts from photos — majidmanzarpour · 2026-11-23
- Dev levels up Neon-built escape room game with an epic AI-generated intro video — jacalulu · 2026-11-23