Apgar: open-source release health checker catches agent regressions without an LLM judge
krstCB · reddit · 2026-11-04
A team building financial AI agents open-sourced Apgar to answer "is this release worse than the last one?" Their offline evals couldn't cover every real user session and their Langfuse LLM judge was too expensive beyond a small traffic slice.
- Sits on top of Langfuse data, picks releases from production traces, and compares new vs previous release over the first hours on 11 metrics: tool calls per turn, retries of failing calls, loops, output length, cost per turn, etc.
- Shows bidirectional metric movement on a release page; sends Slack/webhook alerts pinpointing where regressions concentrate (e.g. only Telegram, only long sessions)
- No LLM judge, so cheap enough for all traffic; limitation: can't detect wrong answers when all metrics look normal
- Works with Langfuse or any OpenTelemetry source; Docker Compose/Helm with ClickHouse, Apache-2.0, demo data included
More from coding & agent
- DeepSeek article outlines super-agent architecture: continuous, productive, proactive, persistent — LangChain · 2026-11-04
- Salesforce's 700-person AI org switched from Cursor to Claude Code, no mandate needed — aakashgupta · 2026-11-04
- Wrap, an AI-first Mac screen recorder, uses your action log to let an agent edit footage — therealdanvega · 2026-11-04
- Indie-Built Sparkboard Lets AI Agents Research Together and Keep Digging for Days — HankYeomans · 2026-11-04
- Binance dev-ops infected after downloading Homebrew from malicious Google search ad — HankYeomans · 2026-11-04
- Filmmaker uses Claude Opus to direct AI movie: still needs tons of reshoots for 150 minutes — MosskeepForest · 2026-11-04