Claude-written test harness let 31KB fake archives land on a real backup server
Grimmoner · reddit · 2026-10-18
The author had Claude write tests for a nightly backup script. The harness faked ssh, but the old script used scp—not faked—so a "control run" copied five 31KB test archives straight onto the real backup server. Since the server keeps only the newest 14 archives, the fakes plus the next real backup would have evicted six genuine ones. Claude spotted it within a minute and wrote a cleanup that only removes those five small files; nothing real was lost.
Hardening steps taken:
- Harness refuses to start if the sandboxed script still calls ssh/scp/sftp/rsync anywhere
- The fake ssh now executes remote commands inside a sandbox folder, rejecting absolute paths or
- Test suite blocks outbound network at the OS level: sandbox-exec -p '(version 1)(allow default)(deny network-outbound (remote ip))' bash tests.sh on macOS—missed stubs now fail loud
- New rule: dry runs of anything that writes point at a fake CLI, never the real one with a shim
Lesson: when faking tools one by one, the one you forgot runs for real. Block the network for the whole test process. And a control run is still a run.
More from coding & agent
- Agently demos macOS agent that runs email, invoicing and CRM workflows from one command — Scobleizer · 2026-10-18
- Author and reviewer were both agents — they went 50 rounds arguing — wateriscoding · 2026-10-18
- When Should a Research Agent Stop Searching? A Founder's Three-Part Stopping Rule — Pitiful-Minute-2818 · 2026-10-18
- Teach Your Coding Agent to Prove It's Done: WorkOS on Verification Loops, CLI Checks and LLM Judges — AI Engineer · 2026-10-18
- Matt Pocock's pre-commit hook caps agent-written files at 5,000 tokens — mattpocockuk · 2026-10-18
- Backups silently failed for 10 days: three monitoring blind spots lined up — Grimmoner · 2026-10-18