Agents Drift in Prod: Dev Builds Provider Twins and Mid-Flight Invariants to Catch It
Common_Dream9420 · reddit · 2026-11-17
The author describes a recurring problem: agents behave fine in dev but drift in production as provider responses shift subtly. Two mitigations:
Provider twins: shadow versions of LLM and API calls that replay recorded real responses while letting you inject edge cases before anything hits production.
Inflight invariants — the more effective technique: instead of post-run output checks, define custom assertions that fire mid-execution, e.g. "if this tool call returns shape X, the next hop must satisfy Y." This catches coordination failures much earlier than post-run evals, especially when two agents share state and one changes something mid-flight.
The author is still working on making invariants portable across workflows without over-hardcoding context, and asks the community: should invariants live at the agent level, task level, or orchestration layer, and how to handle async coordination without clean sequencing?
More from coding & agent
- Chrome lets sites self-label AI safety for agents — who warrants the claim? — YvesMulkers · 2026-11-17
- HTTP 402, Idle for 30 Years, Revived so AI Agents Can Pay on the Spot — 0xsachi · 2026-11-17
- Dodaios: Open-Source Local AI Assistant OS That Conducts Claude Code and Codex Agents — InternationalGap3698 · 2026-11-17
- jshookmcp Ships 736 MCP Tools via Three-Tier Profiles to Cut 109K Token Context Cost — Aggravating_Ring5694 · 2026-11-17
- DevScribe AI Agent Turns a Codebase Into a Full Engineering Workspace for $0.05 — Limp_Celery_5220 · 2026-11-17
- Weekly MCP server security scan grades 264 public servers — only 2 get an F — SarahVoge · 2026-11-17