Agent finished the task, but the caller never got the answer: a restart incident postmortem
jonah_omninode · reddit · 2026-10-02
OmniNode shares a real incident: a task arrived one second before a redeploy restarted services. The agent finished its work 23 seconds later, but the caller timed out after five minutes with no answer.
Two root causes: the request was marked handled before returning, and the shutdown sequence closed the reply connection before flushing queued answers, leaving the caller unable to tell failure from in-progress.
After the fix, graceful restarts return an explicit failure, and hard restarts redeliver the request to completion. But the author warns that redelivery can repeat side effects like file edits or comments, since idempotency checks aren't built yet.
He asks peers how callers should learn a task's outcome when workers restart mid-flight.
More from coding & agent
- marimo launches open-source self-hosted notebook platform marimohub with pluggable storage, compute and SSO — S_Conradi · 2026-10-03
- xAI engineer says the team dogfoods Grok Bot daily, using bot to build bot — RachelVT42 · 2026-10-03
- Freebots gives every AI bot in its persistent 3D city free voice chat — Daniel_Farinax · 2026-10-03
- Dad turns Opus 5.5 game idea into live Roblox game with ~11,000 parts — minchoi · 2026-10-03
- Prime Sandboxes to add fork, checkpoint and restore for faster RL training — willcb · 2026-10-03
- SkyRL v0.4 trains 1T-param Kimi K2.7 with RL on just 16 B300 GPUs — casper_hansen_ · 2026-10-03