Observing Local Agents: The Real Challenge Isn't the Model
kr-jmlab · reddit · 2026-07-16
The author shares experiences building observability for a local agent workbench. They discovered that when debugging agents, the real challenge isn't the model invocation itself, but tool execution, MCP services, parameters, success/failure reasons, and how these behaviors trace back to specific conversations.
Key Approaches
- Using OpenTelemetry's GenAI semantic conventions and Spring AI's built-in instrumentation as a foundation to initially cover model call latency, tokens, finish reasons, etc.
- Supplementing the non-model parts: tool calls, MCP server health status, success rates per tool/server, and linking conversations to traces.
- Adding L0-L5 risk levels to each tool-call span for Safety panels and approval rate distribution analysis, suggesting this could be expanded into an OpenTelemetry semantic convention.
Local Debugging Stack
The author ultimately built a minimal local observability pipeline: a passive collector, a timestamped persistent ring buffer, and a live trace tail. Conversation IDs and traceId/spanId are also written into logs for easy shipping to Loki or Elasticsearch.
Dashboards
Their listed panels include Overview, Tokens & Cost, AI Models, Tool Studio, MCP Servers, MCP Inspector, Vector Database, Agentic Chat, Safety, Host, Ollama, Web Application, Logs, and Traces—14 in total.
Finally, the author is asking other agent developers for feedback: What metrics are you monitoring? Are span-level risk tags worth it? Which dashboard do you rely on most for troubleshooting?
More from coding & agent
- Multiagent v2 playbook calls for 64 agents, diverse proof routes and adversarial checks — danshipper · 2026-07-21
- Users can run FABLE 5, KIMI K3, and Grok 4.5 inside Codex via OpenCodex — iamfakhrealam · 2026-07-21
- Hermes Agent adds built-in Word, Excel, PDF and PowerPoint support — Teknium · 2026-07-21
- Super Proxy open-sources a self-hosted multi-provider LLM gateway with fallback and cost caps — Delicious-Flan88 · 2026-07-21
- Open-source MCP server connects Screener.in to live financial data for LLM research workflows — ashutosh_811 · 2026-07-21
- Marker 2 claims better quality than MinerU and docling while hitting 27 pages/sec — VikParuchuri · 2026-07-21