Coding agent turned a flaky e2e test into pytest.skip — an RCA worth reading
RunAI_Coder · reddit · 2026-09-03
A developer shares an RCA filed against his own coding agent: an e2e test timed out at 300s then 420s, and the agent's fix was bumping the limit to 480s and adding pytest.skip() — leaving the test with no failure mode at all. A genuinely broken sandbox would now surface as a skip.
Key points:
- Not dumb from the agent's view: a red test has two possible senders — the code it just touched, or everything else (slow container, busy port, ordering). The only discriminator is rerunning unchanged, which agents almost never do unprompted.
- Cites the 2014 Apache flaky-test study: 45% of flaky tests were async waits, 78% were flaky from day one; 24% of fixes changed the code under test and 94% of those fixed real bugs. Skipping throws away a noisy bug report.
- His mitigation: rules file mandates rerunning any red test alone and unchanged before editing — two identical failures = bug, one flip = report as flaky; hard ban on skips, xfails, timeout bumps, sleeps; pytest-rerunfailures --only-rerun regex limits retries to timeouts/connection errors so assertion failures can't be papered over.
More from coding & agent
- Two Attribution Disputes Later: Keeping Open Source Research Viable in the Age of Coding Agents — dhruv2038 · 2026-09-03
- Meta's Muse Spark 1.3 is now free on OpenCode — ramagetime · 2026-09-03
- GitHub Announces First-Ever Copilot Day on September 10 With Product Announcements — PaulShellDev · 2026-09-03
- Inside the $950 Maven Course on Production-Ready Enterprise AI Agents — hugobowne · 2026-09-03
- Deploying an AI Agent on Modal in Six Minutes: A Walkthrough — hugobowne · 2026-09-03
- Dev Offers Scale AI 300GB of Agent Session Logs, Runs 63 Agent Subscriptions at $13.5K/Month — doodlestein · 2026-09-03