Humans are Missing from AI Coding Agent Research
Zora Z. Wang, John Yang, Kilian Lieret, Alexa Tartaglini, Valerie Chen, Yuxiang Wei, Zijian Wang, Lingming Zhang, Karthik Narasimhan, Ludwig Schmidt, Graham Neubig, Daniel Fried, Diyi Yang
cs.HC, cs.AI, cs.SE
2026-07-04
CMU, Stanford, Princeton and UIUC researchers argue coding agents' bottleneck has moved from task-solving to interaction: alignment, steerability, verifiability, adaptability.
Coding agent research has converged on a single objective: harder benchmarks, longer horizons, higher success rates. The route works, but this 13-author position paper from CMU, Stanford, Princeton and UIUC argues the marginal returns are shrinking and the bottleneck has relocated. Modern agents produce plausible code; what breaks in practice is everything around it. Users struggle to state intent, outputs are too bloated to review, behavior resists mid-course correction, and lessons evaporate between sessions.
The headline evidence is patch bloat: on SWE-bench Verified, successful patches from all nine evaluated models, from Devstral up to GPT-5.2 High, run 1.2x to 2.0x the length of the human-written gold patch, high scorers included. Longer patches cost more to review, and that is exactly where the gap between "passes the tests" and "usable" opens. Real programming was never one-shot: goals shift and oversight is continuous.
The paper decomposes human-centeredness into four interaction-level dimensions, each with a computable formulation, one per phase of the collaboration loop.
The shared blocker is infrastructure: no open pipeline samples human intent, judgments, and interventions at scale. Proposals include user simulators trained on GitHub interaction data, porting HCI metrics like CUPS and PULSE into training, and accepting that coding agents are already general agents, where tending a greenhouse or running a smart home exercises all four pillars natively.
A position paper runs no main experiments. The supporting numbers:
| Evidence | Number | Setting |
| Patch bloat | 1.2–2.0x gold length | SWE-bench Verified, 9 models |
| AI tool adoption | 84% of developers | Stack Overflow 2025 survey |
| Trust gap | half distrust outputs; two-thirds report heavier debugging from half-baked solutions | same survey |
| Interaction data scale | 400M+ daily accept/reject decisions | Cursor Tab RL |
| Cross-task transfer | 850 same-repo Django tasks, zero accumulation | SWE-bench |
There is no comparison against any baseline, and none of the four formulations has been optimized inside a real system.
For agent builders, this is a diagnostic checklist. Each dimension translates into a product metric: turns needed to align, trajectory change after an intervention, human-gold agreement rate, improvement from the first session to the k-th. Computable means it can enter experiments and training objectives.
For researchers, the agenda is concrete: user simulators, interaction data infrastructure, verification beyond unit tests. Industry is already there. Copilot Arena's in-IDE preference signals diverge substantially from static benchmark rankings, and Cursor's Tab RL mined 400M daily decisions to find that when to suggest matters as much as what to suggest, a signal pass@k cannot see.
This is agenda-setting, not a capability jump.
Acknowledged by the authors: the four pillars may not be exhaustive (safety and proactivity are filed as derivative properties in the appendix); current user simulators behave homogeneously and overly cooperatively, and more prompting will not fix it; the formulations depend on human behavior distributions that no one can sample at scale yet, a chicken-and-egg problem.
Questionable on read: the link from longer patches to worse human review is asserted by intuition, with no measurement; the rebuttal to "AI will write 90% of code", that human effort moves upstream to specification and evaluation, is argument rather than evidence. The direction is probably right. The timeline is anyone's guess.