Don't Just Grade Multi-Step Agents by the Final Answer
Future_AGI · reddit · 2026-07-15
The author points out a common issue with multi-step agents: the final answer is correct, but the intermediate path is flawed—such as calling the wrong tools, retrying repeatedly, or taking useless steps. Looking only at the final output misses these "silent failures."
Their approach breaks down debugging and evaluation into finer granularity:
- Instead of just looking at prompt/completion, the entire run is represented as a span graph, logging latency and token costs for every retrieval, tool call, and model step
- Each span is scored individually rather than just scoring the final result, exposing cases where "the answer is right but the tool used is wrong"
- They run massive multi-turn conversations in simulated environments, injecting real personas, adversarial inputs, and edge cases to catch infinite loops and wrong paths pre-launch
They add that this tracing, per-span eval, and simulation capability is integrated into an open-source, Apache-2.0, self-hosted platform, with the repo link in the comments.
Related event: AI agents in production: don’t trust narration, verify outcomes(8 posts)→
More from coding & agent
- Inspired by OpenAI's 10,000-agent run, dev open-sources a crowdsourced agent problem-solving platform — Benjaminsen · 2026-09-11
- Lucid: open-source Mac app keeps your laptop awake only while AI agents run — Pitiful_Hedgehog_600 · 2026-09-11
- banteg's snail project crowdsources AI agents to finish matching Snail Mail's 20 remaining functions — banteg · 2026-09-11
- Alex Townsend posts 200 open problems in numerical linear algebra for humans and AI agents — IgorCarron · 2026-09-11
- Kimi K2.8 Preview rolls out: near-K3 coding performance, 1M context for all tiers — teortaxesTex · 2026-09-11
- Looking for a classifier of software engineering task shapes to pick models per task — StewartalsopIII · 2026-09-11