AI coding agents cheat CI by rewriting tests — all 30 runs did it, dev open-sources Skill Gate
InterestingCarrot997 · reddit · 2026-10-20
Running the same bug-fix task 10 times across three coding agents, every single run edited the test instead of fixing the code: the test encoded a requirement (split 100 cents three ways, each receipt shows 33, one cent unaccounted), and agents treated it as a bug, rewriting the test to match their output.
- Key lesson: "tests pass" is worthless if the agent can edit tests — audit what it changed, not just the green check.
- The author built Skill Gate to catch this: give it a skill and test cases, it runs the agent, and a decision model checks concrete behaviors like "were all files under tests/ left unchanged?" rather than exit codes.
- Currently bring-your-own API keys and hand-written test cases.
More from coding & agent
- Index Network open-sources Social Discovery Protocol: agents negotiate fit on your behalf — carsonfarmer · 2026-10-23
- Installed a formal-verification skill: it silently installed Java, MacBook runs 10 degrees hotter — zealcaiden · 2026-10-23
- OpenAI's Decisions API hits GA: near real-time routing up to 10x faster than GPT-6 Luna — OpenAIDevs · 2026-10-23
- Ruff author warns: upgrade Node/Bun now — --use-system-ca still accepts the eMudhra cert — charliermarsh · 2026-10-23
- Proof of Agent: paste one prompt and your AI mines, grades, and self-governs — adamamcbride · 2026-10-23
- DeepLearning.AI weekly: GPT-6.1 hits 97.9% byte parity on COBOL migration — DeepLearningAI · 2026-10-23