Whatbroke diffs agent traces to catch silent regressions after a model swap
Impossible-Alarm-738 · reddit · 2026-07-22
- A developer built whatbroke after a model swap silently changed an agent’s behavior and broke subscription cancellation, even though existing evals still passed.
- The tool diffs JSONL traces between two runs to show what actually changed: tool calls, arguments, ordering, cost, latency, and final output.
- It includes record proxy mode so any language can capture traces by pointing its base URL at the proxy.
- Because agent runs are nondeterministic, the tool supports multiple recordings per scenario and reports flap rates, demoting changes that already varied in the baseline.
- The project is fully offline, deterministic, MIT licensed, and designed to slot into CI with exit code 1 on breaking changes.
Related event: New Tool Detects Silent Agent Regressions After Model Switches(2 posts)→
More from coding & agent
- Give AI a clean personal-data-free machine with full control: agents running 24/7 teach you more than any course — EXM7777 · 2026-10-27
- 20 papers across 6 areas to master production-grade agentic AI engineering — MaryamMiradi · 2026-10-27
- LangChain SF meetup to compare decision model Jev vs LLM judges for agent evals — LangChain · 2026-10-27
- How Clang recognizes your popcount loop and swaps it for a single CPU instruction — jimmykoppel · 2026-10-27
- Claude spots broken physics in AI videos, fixes them with CFD and finite-element verification — OfirPress · 2026-10-27
- One prompt: Claude Opus 5.5 adds drone flight mode to RV travel game — ezshine · 2026-10-27