How to stop coding agents from writing tests that pass but prove nothing
ComprehensiveShake76 · reddit · 2026-10-21
A developer's agent setup runs unit tests in its own Linux container and only counts a task done with real output (6 tests, OK, exit 0), but the same agent writes the tests—so weak tests still pass. Community-sourced countermeasures: mutation testing (break code on purpose; if tests still pass they prove nothing), hashing the tests folder before the task and requiring human review of any test diff, failing runs when test counts drop or new skip markers appear, and letting the agent add tests but never silently edit existing ones. Key takeaway: "a green test run is a claim, the test diff is the evidence." Author notes this is a small personal setup, and mutation testing gets slow on larger repos.
More from coding & agent
- Training an original-character LoRA for MiniMax H3: full workflow and pitfalls — DeliciousGorilla · 2026-10-21
- Dev builds open-source Chrome extension unifying Gemini and Antigravity on Windows — iamfivestars · 2026-10-21
- MCP server SDK CVE-2026-111770: shared McpServer leaks responses across clients — Future_AGI · 2026-10-21
- When a Customer's AI Assistant Calls Support, Who Checks What It's Authorized to Do? — IrfanZahoor_950 · 2026-10-21
- Two weeks in SF: fundraising goes binary, top devs now review agents not code — ai · 2026-10-21
- Hugging Face livestream tomorrow: scaling RL and evals to train frontier agents — ben_burtenshaw · 2026-10-21