Prompt scored higher? Model drift may be why — dev open-sources paired-testing tool

gael55x · reddit · 2026-11-07

Developer gael55x highlights a common trap: a higher prompt score doesn't mean your new prompt is better, because models drift silently. Stanford/Berkeley researchers ran identical questions through GPT-4 in March and June 2023 and watched prime-number accuracy fall from 84% to 51% with zero prompt changes. His open-source answer, promptcheckup:

Paired comparison: runs old vs new prompt back-to-back on identical inputs and returns better/worse/not-sure-yet with a 95% interval — background drift cancels out.
Validated by simulation: under weekly drift, naive week-over-week comparison picked the worse prompt 41% of the time; the paired version never erred in 2,000 runs.
CI-friendly: exit codes 0/1/2 mean better/worse/undecided, so a bad prompt fails the build. Works with your own score or an LLM judge that is sanity-checked first; a flaky judge yields "not sure yet" instead of a fake answer.
Zero dependencies: pure Python stdlib, with a no-API-key demo in the repo.

Author notes limits: small improvements need many test cases, and biased cases can't be fixed.

Original post →

Prompt scored higher? Model drift may be why — dev open-sources paired-testing tool

More from coding & agent