Agent loops can quietly turn your eval set into training data

Vegetable-Rub-8241 · reddit · 2026-07-29

Your eval set stops being a true eval set once an agent can optimize against it.

The core failure mode: selection itself becomes a weak form of training. If you run many candidates through the same fixed benchmark and keep the winner, you are partly fitting benchmark noise.
Faster agent loops make the problem worse because they burn through the same held-out set more quickly.
What helped in practice:
Count every trial, including discarded candidates, so deflated scoring is honest.
Reveal eval data in blocks instead of keeping one fixed held-out set.
Put the scorer behind a tool boundary the agent cannot write to; in their case, deterministic code behind MCP servers.
They also dropped a robustness metric because the search learned to game it, and they compare the LLM loop against a non-LLM genetic programming baseline using the same controller and scoring seam.

The post asks a practical question: in any agentic self-improvement loop, what actually prevents the benchmark from quietly turning into a training set?

Original post →

Agent loops can quietly turn your eval set into training data

More from coding & agent