Devs on Reddit debate: what do you actually do after your agent eval catches a failure?
Sensitive-Parsnip-12 · reddit · 2026-09-27
A developer observes that many agent teams already have evals, tracing, monitoring and internal tooling — but asks what happens after those systems flag something wrong. Does the eval get you close to the issue, or just say "it's broken" before you dig manually?
The post lists common investigation moves: reading traces by hand, comparing to a reference run, replaying, inspecting tool args/state/prompts, writing ad-hoc scripts, adding logs, pulling in a teammate.
A personal takeaway: treat run comparisons as another piece of evidence, not ground truth — the reference run has its own variance, so you may chase a difference that means nothing. The focus is on the nasty cases where the agent finishes but the behavior/output is wrong, not obvious crashes.
More from coding & agent
- Diplomacy comes to Multi-Agent Arena: test your social strategy against frontier LLM agents — ycombinator · 2026-09-27
- RowRun launches AI automation engineer that turns weekly processes into agent workflows — paw_lean · 2026-09-27
- Serve models from KitOps ModelKit on HAMi: a registry-native path to SGLang inference — HowDevelop · 2026-09-27
- Google blocks an AI assistant from managing Gmail filters: 'this app is blocked' — altryne · 2026-09-27
- Elicit's new AI-era workflow: 10-minute pre-work chats replace heavy project management — charles_irl · 2026-09-27
- Garry Tan Shares His Favorite Bug-Fixing Workflow: Capy + GStack /autoplan on GPT-6 — garrytan · 2026-09-27