A five-step test design for catching agent policy regressions in refund skills
MiserableFlatworm337 · reddit · 2026-10-23
The author outlines a test design for repetitive policy tasks where an agent's plausible answer can still use the wrong deadline or include an excluded charge. Using a fictional refund policy (30-day window, shipping excluded, cents rounded down), the five steps: write expected outcomes for days 29/30/31 before touching the skill; mix in prices, shipping charges, and rounding-exposing values; add missing-input and policy-contradicting follow-ups; save baseline results and rerun identical cases after revising instructions; check separate holdouts instead of rewriting failing tests.
The key distinction is improving how instructions read versus proving behavior changed on the cases you care about. The post also introduces skillEd, an Apache-2.0 toolkit for versioned cases, agent-driven evals via CLI/MCP, and report comparisons.
More from coding & agent
- Developer uses Gemini as software architect to build Apple Watch caffeine tracker — pythononrailz · 2026-10-24
- Dev builds two-tier context mode into AGENTS.md, promises full open-source release — khademinori · 2026-10-24
- Blinded agents compare formal verification vs code review: empirical results are in, and they're striking — rickasaurus · 2026-10-24
- GitHub ships HydraFusion Max, orchestrating Claude Opus 5.5 and GPT-6 models into one response — unixterminal · 2026-10-24
- kimo: an open-source no-GUI video editor that coding agents drive over MCP — LieTop7965 · 2026-10-24
- tinyfool builds a personal shell app: fold remote output, edit remote files locally with auto-diff — tinyfool · 2026-10-24