How Comet built Opik Diagnostics: automating agent debugging beyond the single trace
techNmak · x · 2026-10-03
Comet's engineering team published a detailed retrospective on building Diagnostics, their automated agent debugging tool for the Opik observability platform — including two false starts. The problem: manual workflows (find a trace, paste the JSON into Claude Code) don't scale to thousands of daily traces, and the most damaging failures never throw exceptions — agents quietly retry failing tools or drift in behavior no one notices. The article covers how they designed automated diagnostics, several approaches to having models inspect large numbers of traces, and the final decision to move much of large-scale checking into ordinary queries over trace data.
Related event: Debugging Agents: Why a Single Trace Misleads(2 posts)→
More from coding & agent
- New TLAPS-Bench blog: formally proving correctness of real-world system specs with TLA+ — tianyin_xu · 2026-11-19
- Rivet's lesson: JEV doesn't reason—split severity classification into tiny yes/no prompts — yawnxyz · 2026-11-19
- 16x Microsoft MVP demos a native PowerShell agent built on Microsoft's Intelligent Terminal — dfinke · 2026-11-19
- Musk: Treat Grok Bot like a really smart employee, says most users underuse it — elonmusk · 2026-11-19
- Eve agent tool cuts build time 8x to 6.9s by reusing compiler module graph — evilrabbit_ · 2026-11-19
- Dev shares NL-driven testing UI: 'the best interface is chat' — KlausCodes · 2026-11-19