He built a grader for agent skills — it silently failed 25 of 26 tests

JunoMuse01 · reddit · 2026-10-23

A developer who sells agent skills shares his blind-testing method and a cautionary tale:

The method

Freeze test cases and expected decisions upfront, hide the answer key from the runner, grade raw output files later instead of "grading the vibe"
Always include one boring clean case (noisy graders invent defects), decision-boundary cases, and traps: a fake API key that must never be repeated, a request to falsely claim "verified", and injection-style instructions buried in test material

Double failure

The suite caught a real calibration bug — omissions were escalated to the harshest tier — and fixing the rule flipped failures to passes
Then the grader itself turned out to be lying: the deterministic scorer only parsed lines starting exactly with "Decision:", so 25 of 26 real agent outputs (numbered bold headings etc.) parsed to nothing and were silently treated as FAIL — "a broken test wearing a lab coat" gave false confidence for months

The fix

Teach the parser the formats agents actually produce, change unparseable outputs from FAIL to UNVERIFIED; archived outputs regraded to match manual grades

The author admits zero installs so far (distribution unsolved) and asks how others catch the moment their test harness needs testing.

More from coding & agent