Don't Just Grade Multi-Step Agents by the Final Answer
Future_AGI · reddit · 2026-07-15
The author points out a common issue with multi-step agents: the final answer is correct, but the intermediate path is flawed—such as calling the wrong tools, retrying repeatedly, or taking useless steps. Looking only at the final output misses these "silent failures."
Their approach breaks down debugging and evaluation into finer granularity:
- Instead of just looking at prompt/completion, the entire run is represented as a span graph, logging latency and token costs for every retrieval, tool call, and model step
- Each span is scored individually rather than just scoring the final result, exposing cases where "the answer is right but the tool used is wrong"
- They run massive multi-turn conversations in simulated environments, injecting real personas, adversarial inputs, and edge cases to catch infinite loops and wrong paths pre-launch
They add that this tracing, per-span eval, and simulation capability is integrated into an open-source, Apache-2.0, self-hosted platform, with the repo link in the comments.
Related event: AI agents in production: don’t trust narration, verify outcomes(8 posts)→
More from coding & agent
- Devin adds e2b sandboxes for remote agent execution — badphilosopher · 2026-07-22
- Hermes Agent Refactoring Proposal: Decoupling via Event Bus and Monorepo Slicing — Promptmethus · 2026-07-22
- ty now reads Pydantic config keywords and field metadata — charliermarsh · 2026-07-22
- Pensar Launches AI Security Agent to Autonomously Discover and Patch 0-Days — andriy_mulyar · 2026-07-22
- ty adds first-class Pydantic support, including strict and lax field handling — charliermarsh · 2026-07-22
- Google launches Gemini 3.5 Flash Cyber for CodeMender, with limited access for governments — GoogleAI · 2026-07-22