Why production AI agents drift after passing every eval
Diligent_Response_30 · reddit · 2026-07-29
Why agents still drift in production after passing every eval
The author, who says they evaluate AI agents in production for a living, describes a recurring failure mode: an agent passes every staging eval, then slowly drifts once real traffic arrives.
The post argues that snapshot-style evals are not enough because production traffic changes over time. The main causes called out are:
- Distribution shift between eval data and real user inputs.
- Upstream prompt or model changes that alter behavior after launch.
- Tool and API response shape changes that agents handle silently.
- Lack of full-run observability, so issues surface only after downstream complaints.
The proposed fix is to treat evaluation as continuous: trace every run, score drift and risk in real time, and keep a human-in-the-loop option to hold or block risky runs instead of only logging them for later review.
The author also notes the team is building in this exact space and is live on Product Hunt.
More from coding & agent
- Hermes Agent Desktop impresses users with parallel tools and remote local-model setup — Teknium · 2026-07-29
- GrokTerm brings Grok Voice, MCP-ready tabs and audio fixes to Linux — Daniel_Farinax · 2026-07-29
- Using iMessage to text Codex from a plane would keep cloud agents alive offline — rudrank · 2026-07-29
- Slack bot using ChatGPT image generation gets the account banned after 12 hours — jbrov19 · 2026-07-29
- How do you stop GPT from generating junk unit tests all the time? — zeeg · 2026-07-29
- Reddit user looks for a more reliable agent to download and organize IMF reports — Joejoe10x · 2026-07-29