Whatbroke diffs AI agent traces to catch tool-call regressions after model swaps
Impossible-Alarm-738 · reddit · 2026-07-25
- A builder noticed that after a model swap, an agent still said “done!” to users but had silently stopped calling cancelsubscription.
- Text diffs were useless for catching the regression, so they built whatbroke, a tool that diffs agent trajectories instead of just text.
- The tool compares two JSONL traces and reports what changed: tool calls, arguments, call order, cost, latency, and final output.
- It supports a proxy recording mode for any language, can import traces from Langfuse/LangSmith/OpenTelemetry GenAI spans, and marks breaking changes with exit code 1 so it can fit into CI.
- The project is MIT licensed, works fully offline, and is on GitHub as arthi-arumugam-git/whatbroke.
Related event: whatbroke: Catching AI Agent Behavior Regressions(2 posts)→
More from coding & agent
- Two real 'company brains' opened up live: Gorgias' in-house Cortex vs Slite — femke_plantinga · 2026-09-11
- The browser main thread is expensive: a practical guide to JavaScript and CSS animation cost — jh3yy · 2026-09-11
- Inspired by OpenAI's 10,000-agent run, dev open-sources a crowdsourced agent problem-solving platform — Benjaminsen · 2026-09-11
- Lucid: open-source Mac app keeps your laptop awake only while AI agents run — Pitiful_Hedgehog_600 · 2026-09-11
- banteg's snail project crowdsources AI agents to finish matching Snail Mail's 20 remaining functions — banteg · 2026-09-11
- Alex Townsend posts 200 open problems in numerical linear algebra for humans and AI agents — IgorCarron · 2026-09-11