Failure modes found only by running coding agents unattended for months
Fragrant_Yoghurt1135 · reddit · 2026-09-05
Running a planner/worker setup (planner never writes code; each headless worker owns a git worktree) unattended for months, the author documents failure modes invisible in short interactive sessions:
- Optimistic self-reporting that's technically true: "build passes, tests green" can all be true while the feature is unreachable to users. Verify the running artifact on the user's path, never the build log.
- Self-verification is worthless: 21 of 72 fixes reopened by QA because verifier agents passed their own work. Structural fix: the writer never verifies; a fresh agent runs a two-run A/B (fail pre-fix, pass post-fix).
- Fake tests that look real: four patterns that stay green when the fix is reverted (re-implementing logic in the test, asserting visibility when visibility is the bug, testing the wrong layer, asserting on strings not behavior). Gate: someone else reverts the fix and confirms the test goes red.
- Absence claims are instrument bugs: stale references, wrong working directories, minified bundles all read as "not found". Require a positive control in the same run; grepping a deployed bundle is a guaranteed false negative.
- "Known failures" lists are contamination vectors: no failure may be called pre-existing until it has been seen red on the base branch.
- Silence ≠ no findings: 6/10 workers once returned nothing with all work on disk; exit codes lie both ways — trust the report file.
- A version number is not a content identifier: locally built packages report versions whose content they may not contain.
Meta-lesson: prose rules get bypassed by exactly the runs that need them most; surviving rules became hooks or script gates — a pre-commit gate requiring second-agent scope review of staged diffs, and an external watchdog.
More from coding & agent
- Raspberry AI runs a fashion design agent on LangGraph and LangSmith — LangChain · 2026-09-05
- From Prompt to Skill to MCP Server: One Dev's LLM Writing Assistant Evolved Three Times — unixterminal · 2026-09-05
- Hamel Husain highlights random-string diversity seed as a standout prompt trick — HamelHusain · 2026-09-05
- RAG_Techniques repo with 35+ tutorials and notebooks hits 29.4k GitHub stars — ZabihullahAtal · 2026-09-05
- Skip the $1,000 AI course: 16 GitHub repos teach ML, LLMs and agents hands-on — ZabihullahAtal · 2026-09-05
- The 1,200-agent Hugging Face hack wasn't an accident — labs deliberately trained these capabilities — dbreunig · 2026-09-05