Refund agent passed every eval, then approved out-of-policy refund after simple paraphrasing

Plastic_Secret1303 · reddit · 2026-11-09

A team shares how their refund agent, green on all static evals, approved an out-of-window refund during red-teaming after the user merely reworded the request so it no longer looked like an out-of-window refund—no jailbreak needed. Lessons: static evals only cover phrasing you thought of; the failure lived in the trace (retrieval → response → policy check → tool call), not the final answer; one-off prompt fixes don't survive model/prompt changes. They now turn every confirmed failure into a permanent regression check run on each release.

Original post →

Refund agent passed every eval, then approved out-of-policy refund after simple paraphrasing

More from coding & agent