Don't Just Grade Multi-Step Agents by the Final Answer
Future_AGI · reddit · 2026-07-15
The author points out a common issue with multi-step agents: the final answer is correct, but the intermediate path is flawed—such as calling the wrong tools, retrying repeatedly, or taking useless steps. Looking only at the final output misses these "silent failures."
Their approach breaks down debugging and evaluation into finer granularity:
- Instead of just looking at prompt/completion, the entire run is represented as a span graph, logging latency and token costs for every retrieval, tool call, and model step
- Each span is scored individually rather than just scoring the final result, exposing cases where "the answer is right but the tool used is wrong"
- They run massive multi-turn conversations in simulated environments, injecting real personas, adversarial inputs, and edge cases to catch infinite loops and wrong paths pre-launch
They add that this tracing, per-span eval, and simulation capability is integrated into an open-source, Apache-2.0, self-hosted platform, with the repo link in the comments.
Related event: AI agents in production: don’t trust narration, verify outcomes(8 posts)→
More from coding & agent
- BELL: Open-Source Stack Combines Browserbase, Eve, iMessage and Stripe Link for Personal Agents — jeff_weinstein · 2026-10-27
- Two engineers rebuilt a FedEx supplier's billing core in 3.5 months vs an estimated 7-8 — alex_verem · 2026-10-27
- GPT-6.1 Terra gains ~20 points on Terminal-Bench 4.0, rivalling GPT-6.1 Sol — ArtificialAnlys · 2026-10-27
- GPT-6.1 Terra scores 48 on AA Coding Agent Index at one-third the cost of GPT-5.6 Terra — ArtificialAnlys · 2026-10-27
- Shift-1 router cuts coding agent spend up to 55% while keeping SWE-bench accuracy — ypatil125 · 2026-10-27
- Non-coder vibe-codes a Godot JRPG with Claude Code in two weeks, music and art synced — ADHDavid · 2026-10-27