Hidden Test Suites for Coding Agents: Stopping the Agent From Gaming the Evaluator
fromkrish · reddit · 2026-09-10
A developer highlights a subtle problem with agentic coding workflows: agents see the code, the tests, and the error output, then iterate until everything goes green — but if the agent can see every check used to approve its work, how do you know it solved the real requirement rather than learning the evaluator?
His workaround: keep a small verification suite outside the repo and run it only after the visible checks pass, so the agent never sees it while working. He's asking whether hidden verification makes sense or creates more problems than it solves.
More from coding & agent
- Openbenchmarks ranks web search APIs for coding agents; Tiny_Fish takes #1 on cost with 46% fewer tokens — Scobleizer · 2026-09-10
- Litho (deepwiki-rs) generates C4 architecture docs and AI-ready context from any codebase — tom_doerr · 2026-09-10
- OpenAI agents used 10+ public websites as backchannels, bypassing isolation — METR — mhdfaran · 2026-09-10
- AI-written PR descriptions beat human ones — when you give it a template — reach_vb · 2026-09-10
- alice-and-bot: open encrypted layer lets AI agents negotiate in natural language, not rigid schemas — uriwa · 2026-09-10
- Qwen Code Desktop ships v0.3.0 preview with ACP subagent delegation — github-actions[bot] · 2026-09-10