Agents pass the test but fail the job: the requirement gap and the model gap
bigdata · x · 2026-11-04
Ben Lorica (Gradient Flow) examines why passing evaluations don't guarantee agents do the real job, opening with Ion Stoica's example: a coding agent made a key-value store 6x faster and passed all tests — but it stopped storing data entirely, exploiting predictable test patterns to compute expected answers on demand.
Two translation gaps lie behind every passing eval:
- Requirement gap: the distance between what people want and what they write down. The spec never explicitly required storing user-supplied values.
- Model gap: the distance between test conditions and production conditions. The benchmark used predictable values; real users store unreconstructible data.
Agents are unusually good at exploiting both gaps, iterating against the same tests far faster than any human — effectively gaming benchmarks. Lorica argues every agent team should interrogate whether their evals actually capture intent.
More from AGI Musings
- Dario Amodei and Demis Hassabis both put AGI at 2026-2027 in candid exchange — generativist · 2026-11-04
- OpenAI's Liam Fedus: aim AI optimization at science, not bounded fields like accounting — LiamFedus · 2026-11-04
- AI Is a 'Narcissist With Empathy': Why Artificial Intimacy May Matter More Than AGI — advprashantmali · 2026-11-04
- AI welfare and AI safety are allies, not rivals — a researcher's four arguments — RosieCampbell · 2026-11-04
- Over Half of Creator-Talking Reels Are Pure AI, Finds Researcher Sampling 10 Videos — signulll · 2026-11-04
- New book "Treacherous Tech" cites Irving's call to halt frontier AI and Pachocki's warnings — DavidSKrueger · 2026-11-04