252/252 Mutants Killed—and Every Kill Was Fake: A Mutation Testing Postmortem
Grimmoner · reddit · 2026-10-19
The author uses Claude Code to build small Python home-automation tools and runs mutation testing on anything that matters: deliberately break the code one small change at a time and confirm some test fails for each mutant. One run looked perfect—three hours, 252 mutants, 252 killed—until a post-run control check failed and voided everything.
Root cause: the runner copies code and tests into a fresh temp folder per mutant. Three days earlier, a test had started reading a data file next to the code, and nobody added it to the copy list. Every copy—mutated or not—failed that same test, so every mutant "died" from the missing file. The baseline ran on the source folder where the file exists, hiding the bug. The same flaw had already voided an earlier "11 of 11 killed" result sitting in the changelog as evidence.
An earlier incident: a run crossing midnight lost its /tmp test venv config (likely macOS tmp cleanup); pytest couldn't start, and a suite that can't start fails—so everything after midnight read as killed.
Current safeguards:
- Every run includes a comment-only control mutant that must survive; if it dies, the run is void
- The control goes through the runner's own copy step—source-folder baselines hide missing files
- A kill isn't evidence without the failing test name; one test name in every kill means one broken test, not a strong suite
- New file dependencies go on the copy list in the same change
- Long runs are wrapped with source-folder fingerprinting (names, sizes, hashes); any drift marks the run VOID—Claude once added a test mid-run, so later mutants ran against different tests
The lesson generalizes to evals and LLM judges: run one must-pass and one must-fail case through the exact same harness every run. 100% should make you suspicious, not happy.
More from coding & agent
- Xaira launches X-Scientist AI drug discovery agent: nominates targets ~10x faster than humans — BoWang87 · 2026-10-23
- 1Password launches Agents platform to let AI agents work without seeing your passwords — testingcatalog · 2026-10-23
- Cognition open-sources Devin agent plugin to patch the day's CVE wave — Scobleizer · 2026-10-23
- Fine-tuning models solo with a Grok bot and rented GPUs — Rasmic · 2026-10-23
- One prompt found pre-auth PHP RCEs: agent-guided vuln hunting timeline — chrisrohlf · 2026-10-23
- AutoRouter launches beta: auto-routes each coding task across Claude, GPT and other models to cut token spend — floguo · 2026-10-23