TestSlop catches off-by-one 'evil twins' that pass your agent's test suite
chickpo · reddit · 2026-09-25
The author set out to catch coding agents weakening tests to fake green runs—didn't happen often enough. What kept appearing instead: a wrong implementation one edit away passing the same suite. Example: quantity <= 0 vs <= 1, both 3/3 passing, with quantity=1 uncovered.
TestSlop (github.com/hyukvoid/TestSlop) takes an agent's diff, writes one nearby variant (a boundary shifted by one), reruns your suite against it in a scratch copy, and prints both results plus the separating input—source tree untouched. Real case included: replaying the vercel/ms month-formatting commit against its original suite, both pass 163/163, the gap being exactly 1y vs 12mo.
Limits: JS/TS boundary comparisons on changed lines only; no Twin found ≠ correct, and it's not a replacement for mutation testing.
More from coding & agent
- franken_tts license explicitly bars OpenAI and its agents, forcing a switch to a local speech engine — BLUECOW009 · 2026-09-25
- Opus 5.5 worked 8 hours unattended, then re-implemented code that already existed — Sauers_ · 2026-09-25
- Running 15 px0 instances takes only ~320 MiB, less than one zed worktree — arpit_bhayani · 2026-09-25
- A GPT agent reads a project's license and refuses to install it—licensing now blocks AI agents — BLUECOW009 · 2026-09-25
- Skill prompt collection updated, also works well with the zed editor — op7418 · 2026-09-25
- TaskView: Self-Hosted Task Management Platform Hits 732 GitHub Stars — tom_doerr · 2026-09-25