Reconstructing what actually happened when your agent goes wrong: an observability playbook

felix_baron · reddit · 2026-09-25

The author proposes an open-source observability setup for agents, tackling cases like an MCP tool call timing out while the agent claims success.

Key points:

Task-centric tracing: persistent business-task IDs linking multiple sessions, runs, and reviews; record agent version, model, and tool config
Observe tool calls at both ends: distinguish proposed → approved → sent → tool-reported → independently confirmed; timeouts stay "outcome unknown" — blindly retrying writes can do more damage
Separate transport errors, tool-level errors, and answer verification: a 200 response doesn't prove the tool worked, and a successful call doesn't prove the answer is grounded; verify final state in the target system
Three kinds of "replay": browser replay, execution playback, and re-execution (hardest, potentially dangerous — never let a replay silently repeat real payments or DB writes)
Reuse existing stacks: Grafana Faro, OpenTelemetry, Alloy, Tempo, Loki, Prometheus, optionally rrweb; sample routine traces but retain authoritative outcome records separately

More from coding & agent