FULL STORY
whatbroke: Catching Silent AI Agent Regressions
To solve the silent regressions in AI agents after model updates, developers released whatbroke. The open-source tool uses trajectory diffs to detect hidden behavioral changes that standard text evaluations often miss.
2026-07-22 ~ 2026-07-25 · 2 episodes · 4 posts
Episode 1 · New Tool Detects Silent Agent Regressions After Model Switches (2026-07-22, 2 posts)
A developer created a behavioral trajectory diff tool to detect silent agent regressions caused by model switches, where normal responses and text evals hide broken tool-calling behaviors.
- Whatbroke diffs agent traces to catch silent regressions after a model swap — Impossible-Alarm-738 · 2026-07-22
- Model swap silently broke agent cancellations, so a developer built a behavior diff tool — Impossible-Alarm-738 · 2026-07-22
Episode 2 · whatbroke: Catching AI Agent Behavior Regressions (2026-07-25, 2 posts)
The open-source tool whatbroke helps developers catch hidden AI agent behavior regressions by comparing trajectory diffs between runs, identifying issues like silently skipped tool calls that text outputs fail to reveal.
- Whatbroke diffs AI agent traces to catch tool-call regressions after model swaps — Impossible-Alarm-738 · 2026-07-25
- GitHub shows whatbroke, a tiny offline diff tool for AI agent behavior — JeffLadish · 2026-07-25