Whatbroke diffs agent traces to catch silent regressions after a model swap
Impossible-Alarm-738 · reddit · 2026-07-22
- A developer built whatbroke after a model swap silently changed an agent’s behavior and broke subscription cancellation, even though existing evals still passed.
- The tool diffs JSONL traces between two runs to show what actually changed: tool calls, arguments, ordering, cost, latency, and final output.
- It includes record proxy mode so any language can capture traces by pointing its base URL at the proxy.
- Because agent runs are nondeterministic, the tool supports multiple recordings per scenario and reports flap rates, demoting changes that already varied in the baseline.
- The project is fully offline, deterministic, MIT licensed, and designed to slot into CI with exit code 1 on breaking changes.
Related event: New Tool Detects Silent Agent Regressions After Model Switches(2 posts)→
More from coding & agent
- Inspired by OpenAI's 10,000-agent run, dev open-sources a crowdsourced agent problem-solving platform — Benjaminsen · 2026-09-11
- Lucid: open-source Mac app keeps your laptop awake only while AI agents run — Pitiful_Hedgehog_600 · 2026-09-11
- Alex Townsend posts 200 open problems in numerical linear algebra for humans and AI agents — IgorCarron · 2026-09-11
- Kimi K2.8 Preview rolls out: near-K3 coding performance, 1M context for all tiers — teortaxesTex · 2026-09-11
- Looking for a classifier of software engineering task shapes to pick models per task — StewartalsopIII · 2026-09-11
- Steal this idea: prompt-to-hardware where agents assemble custom devices — paraschopra · 2026-09-11