A silent watcher race dropped agent logs for a month: full post-mortem of a tailing bug
ClaudeCdGuy · reddit · 2026-09-22
A developer recounts a month-long silent data-loss bug in his agent observability setup: agents write JSONL transcripts, a hub tails them by byte offset and streams events to a viewer. CI flagged one intermittent failure weekly — always on the busiest matrix leg — and he twice closed it as flaky, even writing "timing budget too tight" in a test comment.
The mechanism: watchers arm asynchronously, and between requesting the watch and the poller taking its baseline stat there's a window. The append landed 2ms after the watch opened, inside the baseline, so the file "never changed" — 26 sweeps over 8 seconds, zero bytes, no error. Slow eventually arrives; lost never does.
Why it never recovered: his 500ms sweep rescanned subagent dirs but not the main transcript, and the live-session sweep returned early. The only backstop fired once on watcher ready — already spent. The fix re-reads the main transcript on the same tick as the rescan (a no-op read costs one statSync), with a deterministic regression test pinning the watcher to one poll per minute.
Takeaways: treat watchers as best-effort and give every tailed file a periodic floor, converting "silently lost" into "at most one interval late"; and intermittent-on-the-busiest-runner failure describes a race, not a reason to retry.
More from coding & agent
- Debate: will all mainstream software be formally verified by 2030? Legacy debt says no — luislamb · 2026-09-22
- OpenMuse: open-source AI coworker that browses the web in a disposable Linux desktop — aniketmaurya · 2026-09-22
- Turn detection experiment: Jev hits 16/16 while tiny-BERT Laya flunks on own data — tobowers · 2026-09-22
- What engineers check before adding a new LLM provider — Rama_Surasani_ · 2026-09-22
- Builder: AI productivity gains in software are 'truly unbelievable' — omnivaughn · 2026-09-22
- OpenAI case study: V7 Go gives enterprise agents long-term memory via Context Graph — xiaohu · 2026-09-22