Hand-scoring my AI action checker: most checks were wrong more often than right

chich2017 · reddit · 2026-11-19

A developer followed up on their AI checker that flags when an assistant claims it did something but didn't — by hand-scoring its checks, with humbling results:

The trap: every fix invalidated the accuracy history, since it belonged to the old version — constant fixing with no number to stand behind.

The fix: freeze the two verdict-deciding checks, with a test that fails if their code changes before being re-scored on 30 real panels at that exact version (currently 0/30, honestly). Shaky checks still run as info only. The author plans a pass using only the original ask and final artifact, and asks: how do you check your checker?

Original post →

More from coding & agent

coding & agent channel →