Backups silently failed for 10 days: three monitoring blind spots lined up
Grimmoner · reddit · 2026-10-18
The author's self-hosted nightly backup failed three nights in a row and no one knew for 10 days. Three separate mistakes lined up:
- The ping never arrived: the ship job pinged Healthchecks with curl and swallowed errors, but the machine couldn't reach the Healthchecks host at all. Healthchecks alerts on state flips, not while a check stays down—a check red for a week is as blind as no check. Fix: the ping now rides the ssh connection used for the copy.
- "Already shipped" counted as success: when the backup broke, the ship job saw the last good archive already on the remote box and pinged green anyway—it checked the newest file was shipped, not that it was new. Fix: fail if the newest archive is over 26 hours old.
- Evidence written by the thing being checked: file names, dates and log "OK" lines come from the same account doing the backup. Fix: open the archive, read the database inside, and require the newest record to be within 6 hours of the claimed time.
All of this was found during an unrelated security review—no monitor caught it. Rule: a green check must prove the thing is new, and the proof can't come from the thing being checked.
More from coding & agent
- Author and reviewer were both agents — they went 50 rounds arguing — wateriscoding · 2026-10-18
- When Should a Research Agent Stop Searching? A Founder's Three-Part Stopping Rule — Pitiful-Minute-2818 · 2026-10-18
- Teach Your Coding Agent to Prove It's Done: WorkOS on Verification Loops, CLI Checks and LLM Judges — AI Engineer · 2026-10-18
- Claude-written test harness let 31KB fake archives land on a real backup server — Grimmoner · 2026-10-18
- Matt Pocock's pre-commit hook caps agent-written files at 5,000 tokens — mattpocockuk · 2026-10-18
- Author Laurence Moroney: write your ugly first draft yourself, then let an LLM edit it — lmoroney · 2026-10-18