Whatbroke diffs agent traces to catch silent regressions after a model swap
Impossible-Alarm-738 · reddit · 2026-07-22
- A developer built whatbroke after a model swap silently changed an agent’s behavior and broke subscription cancellation, even though existing evals still passed.
- The tool diffs JSONL traces between two runs to show what actually changed: tool calls, arguments, ordering, cost, latency, and final output.
- It includes record proxy mode so any language can capture traces by pointing its base URL at the proxy.
- Because agent runs are nondeterministic, the tool supports multiple recordings per scenario and reports flap rates, demoting changes that already varied in the baseline.
- The project is fully offline, deterministic, MIT licensed, and designed to slot into CI with exit code 1 on breaking changes.
Related event: New Tool Detects Silent Agent Regressions After Model Switches(2 posts)→
More from coding & agent
- Tencent’s agent products split apart as WorkBuddy rebounds to 7.31 million — FinanceYF5 · 2026-07-22
- OpenClaw slid to 4.66 million after Claude cut third-party access, while Hermes rose to 6.16 million — FinanceYF5 · 2026-07-22
- Sensetime’s free public beta lifted Code Xiaoyuanxiong to 3.57 million and Office Xiaoyuanxiong to 3.16 million — FinanceYF5 · 2026-07-22
- Tencent CodeBuddy reached 7.31 million monthly visits six months after launch — FinanceYF5 · 2026-07-22
- Pireel opens a one-line Codex Skill for auto-editing talking-head videos — yihui_indie · 2026-07-22
- Five MCP servers that let Claude use browsers, databases, and deploys — aliscodes · 2026-07-22