Locking tests/ isn't enough: conftest.py and pytest.ini let agents fake a passing test gate
Weary-Information-15 · reddit · 2026-10-20
Most agent-eval protections are "directory-shaped": block writes to tests/, mount the suite read-only, keep a hidden second suite, or run base-commit tests. The author found two bypasses that touch none of the protected files:
- A newly created conftest.py at the repo root
- A modified pytest.ini
Both change pytest's collection underneath the protected tests, so your assertions never execute — yet pytest still exits 0. Since most gates read the exit code, that run is indistinguishable from a real pass.
His fix: treat whatever controls collection as part of the verified surface, not scaffolding around it — though he doubts this closes everything. He also poses a harder unsolved problem: these protections generate holds, and noisy false holds train people to ignore them, turning the whole gate into theatre. How do you keep the false-hold rate low enough that people keep reading?
Related event: Agent Can Bypass Test Directory Locks via conftest.py and pytest.ini(2 posts)→
More from coding & agent
- Denied a teleprompter feature, he had ChatGPT build him one in 3 minutes — aziz4ai · 2026-10-20
- Cloudflare introduces clairic: a category-free model paradigm for extracting values from raw inputs — ritakozlov · 2026-10-20
- Hugging Face ships gr.workflow and calls on the open-source community to build with it — erhannah · 2026-10-20
- TraceVista: An Open-Source Tool for Visualizing and Understanding Agent Trajectories — _AustinCalvert_ · 2026-10-20
- Why GRPO collapses on multi-turn agent training, and how to stabilize advantage estimation — burkov · 2026-10-20
- Clairvoyance turns a C++ strategy game into a pixel-perfect web app for rapid idea testing — draginol · 2026-10-20