Do We Use Bootstrap for Agent Evaluations Too?
abcdedcbaa · reddit · 2026-07-19
The post asks: When conducting agent evaluations, should we use large test sets plus statistical sampling, similar to chatbot evals?
The author's current practice is to evaluate with about 70 golden cases, then expand to around 1k+ cases via bootstrap / resampling as a sanity check. They want to know if others do the same for agent scenarios, and whether current agent eval / observability standards have shifted towards trajectory-level metrics, such as tool call sequences, reasoning processes, and final answers.
More from coding & agent
- Spring AI 2.1.0-RC1 adds tool-call confirmation, pausing agent loops before real-data access — therealdanvega · 2026-10-26
- 250 Builders Hack With Codex at Berlin Community Hackathon Ahead of OpenAI DevDay — paw_lean · 2026-10-26
- Building AI bots on raw APIs? Devs say models are 'dumb without the harness' — madeofAjala · 2026-10-26
- HarnessMM cuts agent tokens per task by up to 88% while raising success rates — omarsar0 · 2026-10-26
- Microsoft demos intent-classifying agents built with Copilot App and Foundry, deployable across Teams and Outlook — marlene_zw · 2026-10-26
- Looking for a wearable mic with best battery life that feeds recordings to personal agents via MCP — altryne · 2026-10-26