Whatbroke diffs AI agent traces to catch tool-call regressions after model swaps
Impossible-Alarm-738 · reddit · 2026-07-25
- A builder noticed that after a model swap, an agent still said “done!” to users but had silently stopped calling cancelsubscription.
- Text diffs were useless for catching the regression, so they built whatbroke, a tool that diffs agent trajectories instead of just text.
- The tool compares two JSONL traces and reports what changed: tool calls, arguments, call order, cost, latency, and final output.
- It supports a proxy recording mode for any language, can import traces from Langfuse/LangSmith/OpenTelemetry GenAI spans, and marks breaking changes with exit code 1 so it can fit into CI.
- The project is MIT licensed, works fully offline, and is on GitHub as arthi-arumugam-git/whatbroke.
Related event: whatbroke: Catching AI Agent Behavior Regressions(2 posts)→
More from coding & agent
- Kolab Benchmark Uses Pro Screen Recordings to Show AI Agents Overestimate Collaboration Quality — minnesotanlp · 2026-10-27
- Dev argues Claude burns far more tokens than GPT models for the same coding work — haider1 · 2026-10-27
- Stop building agents that can't change models: stable contracts, external state, real evals — rseroter · 2026-10-27
- Cheap models can cost more in agent harnesses than frontier ones, researcher warns — vasuman · 2026-10-27
- Kanzo: A 5k-Line Rust Kanban Binary With MCP for Claude Code and Codex — allmudi · 2026-10-27
- 'A human approves everything' is a trap: approval fatigue and how to actually design safe agents — Efistoffeles · 2026-10-27