AgentSeism: open-source statistical CI for deciding when an agent truly regressed
puppy_lover_2021 · reddit · 2026-09-27
A developer building CI for agents shares hard-won lessons: an unchanged agent fluctuated from 92% to 88% success (a false alarm if blocked), while a deliberately degraded agent dropped 92% to 52% yet passed a naive statistical rule because damage concentrated in a few tasks. He argues agent PRs should be treated as experiments, not deterministic tests, separating broad reliability drift from single-capability collapse. The open-sourced AgentSeism uses repeated runs, practical effect thresholds, uncertainty, multiplicity across capabilities, and an explicit "insufficient evidence" state. Blog and repo included, seeking feedback on merge-blocking criteria.
More from coding & agent
- Ops optimization for an AI staff team: human bandwidth is the bottleneck — RachelVT42 · 2026-09-27
- MongoDB launches Agent Skills to help AI coding assistants write better database code — TheTuringPost · 2026-09-27
- GitHub's Ash: agents are running a DoS attack on code reviews — jonmarkgo · 2026-09-27
- Using Codex Computer Use to monitor Claude Code's progress end-to-end — vista8 · 2026-09-27
- Claude-generated Three.js 3D harbor scene wows devs with its detail — EricBuess · 2026-09-27
- Agent monitors flight cancellations for pennies using Mercator travel API in Grok Bot — jeff_weinstein · 2026-09-27