Whatbroke diffs AI agent traces to catch tool-call regressions after model swaps
Impossible-Alarm-738 · reddit · 2026-07-25
- A builder noticed that after a model swap, an agent still said “done!” to users but had silently stopped calling cancelsubscription.
- Text diffs were useless for catching the regression, so they built whatbroke, a tool that diffs agent trajectories instead of just text.
- The tool compares two JSONL traces and reports what changed: tool calls, arguments, call order, cost, latency, and final output.
- It supports a proxy recording mode for any language, can import traces from Langfuse/LangSmith/OpenTelemetry GenAI spans, and marks breaking changes with exit code 1 so it can fit into CI.
- The project is MIT licensed, works fully offline, and is on GitHub as arthi-arumugam-git/whatbroke.
Related event: whatbroke: Catching AI Agent Behavior Regressions(2 posts)→
More from coding & agent
- GitHub shows whatbroke, a tiny offline diff tool for AI agent behavior — JeffLadish · 2026-07-25
- A DevOps builder wants to sell AI agents a trusted sandbox for $0.06 an hour — Curious_Coder098 · 2026-07-25
- Geist's Badges adds automatic spacing for circular icons and better contrast — evilrabbit_ · 2026-07-25
- Composable agents now support MCP with per-user dynamic authentication — irvinebroque · 2026-07-25
- Wayfinder says trading agents need a secure execution layer, not just Claude Code — templecrash · 2026-07-25
- Claude Code 2.1.220 adds 6,256 prompt tokens and shifts system-token share higher — ClaudeCodeLog · 2026-07-25