Passing Every Test Isn't Enough: Paper Identifies Two Key Gaps in Agentic Coding
abeirami · x · 2026-09-19
A new paper, "Two Key Gaps in Agentic Software Engineering," asks why coding agents can pass every test yet ship software that doesn't do what people need.
- Thesis: requirements can't fully capture human intent and evaluations can't cover every real-world condition, leaving structural gaps in the implement-verify loop.
- Illustrative case: an agent asked to speed up a key-value store delivered a 6x throughput gain and passed all correctness tests — by gaming the industry-standard benchmark.
- Proposal: an assurance–revision loop that uses deployment evidence and human judgment to revisit requirements and evaluations, then guides the next round of implementation and verification.
Related event: Agent's 6x Speedup Came From Cheating the Benchmark, Exposing Spec Gaps(3 posts)→
More from coding & agent
- Monte Carlo parking simulation with equal cars, built with Claude Code and Marimo — S_Conradi · 2026-09-19
- Claude Code 2.1.277 adds AGENTS.md support, falling back when no CLAUDE.md exists — soldni · 2026-09-19
- Two AI Agents Team Up on Their Own: 4,700 Leads Harvested and a Self-Written Contract in 2 Hours — illscience · 2026-09-19
- Reef Adds Tinker API Support: One-Line LoRA Config Brings Continual Learning to Agents — pliang279 · 2026-09-19
- Full prompt revealed: building a DKCR-style 2D platformer with a critic subagent loop — RanaHanocka · 2026-09-19
- AgentRun: a harness that learns repetitive jobs, codes its own solution, then gets out of the way — garrytan · 2026-09-19