Job queue stuck for two days: the case for leases over a two-state pending/done flag
Kodenstudio · reddit · 2026-10-14
A developer shares a two-day debugging lesson: his job queue had only two states, pending and done. When a worker crashed mid-job, nothing flipped the row, so the next worker picked up the same poisoned task and crashed again — the queue went silently dead while the dashboard looked healthy.
The fix was small: stop trusting the flag and give each job a lease — claim it, stamp an expiresat, send heartbeats while working, and let a simple reaper sweep expired leases back into the pool. "Two hours to build, two days to realize I needed it." He asks whether others run leases on top of a status column or skip the state machine entirely.
More from coding & agent
- 10 GitHub repos revealing how Cursor, Claude, Devin and other AI tools are prompted — victor_explore · 2026-10-14
- Dimillian details building Dark Veil game and custom engine with Codex — Dimillian · 2026-10-14
- SAP paper teaches LLM agents to predict their own failure (AUROC 0.85) — SAP · 2026-10-14
- Your code comments are agent bait: delete most of them, dev argues — weswinder · 2026-10-14
- CMU paper: agent logs prove 0 of 4 violations; structured records raise detection to 83% — rohanpaul_ai · 2026-10-14
- Gemini 4 Argon pattern: server-side background execution for minutes-long reasoning runs — rseroter · 2026-10-14