How to catch prompts that silently degrade when the model version changes
SnooDoubts4271 · reddit · 2026-09-15
A developer running agents in production flags a gap: CI can verify prompt templates are structurally valid, but can't catch templates that still pass yet quietly produce worse output on newer models.
Their proposed approach:
- A few pinned eval cases per prompt template
- A matrix tracking which models each template was actually tested on
- A failing check when scores drop below pinned thresholds
The thread also asks the community: per-prompt evals vs. whole agent runs? How to score free-text outputs without LLM judge drift? Re-run evals on every model update or on a schedule?
More from coding & agent
- tldraw's AI design sprint: Codex prototypes every discussed interaction overnight, artifacts by day 2 — max__drake · 2026-09-15
- Research agents log what they cite — should they also defend what they reject? — Entire_Mark8010 · 2026-09-15
- Grok Bot Auto-Schedules Pinterest Posts, Writes Titles From Images—Where Claude Failed — prasenx · 2026-09-15
- Amplitude tripled PR volume in six months by fixing CI, not agents — mobileraj · 2026-09-15
- Looking for a minimal, near-instant CLI coding agent for one-off bash tasks — funbike · 2026-09-15
- LangChain's Chase on why agent memory never sticks: deciding what to remember is app-specific — blaizedsouza · 2026-09-15