Keeping LLM agents alive on multi-day jobs: heartbeat boards, epoch fencing, and open problems

milkygirl21 · reddit · 2026-10-10

The author runs a fleet of LLM agents on jobs lasting hours to days (e.g. batch transcribing and summarizing hundreds of videos into a knowledge base). The core pain: agents die quietly without crashing. Nine practices already in place:

Shared heartbeat board: one row per job with six standard fields (last check, next expected, deadline, current step, checkpoint link)
Deadlines at 2-3x the check interval — a missed beat is a flag, not a verdict
Bounded retries with caps and total deadlines; standing jobs get no deadline
Parent audit agent every 30 minutes; if over half the jobs are flagged it sends one grouped alert (usually an outage, not ten dead agents)
"Unknown" is never "dead": throttled status checks never trigger relaunches
Intent/done records around irreversible side effects so relaunched agents can tell what already happened
Epoch fencing: relaunched agents get higher epochs; old ones self-terminate on seeing a newer one
Drive-first checkpoints after each batch
A watchdog verifying the audit itself is alive

Open problems: credential delivery (vault only fills browser inputs, so git commits go through a slow browser route), workspace resets wiping local state, no external dead-man switch if the audit and board die together, step/context limits causing dirty stops, and Sheets quota throttling. The author asks for suggestions on all of these.

Original post →

Keeping LLM agents alive on multi-day jobs: heartbeat boards, epoch fencing, and open problems

More from coding & agent