Novice Developers Produce Larger Review Overhead for Project Maintainers while Vibe Coding
Syed Ammar Asdaque, Imran Haider, Muhammad Umar Malik, Maryam Abdul Ghafoor, Abdul Ali Bangash
cs.SE
2026-02-27
Analyzing 22,953 PRs from 1,719 vibe coders in the AIDev dataset, a LUMS team finds the low-experience group submits 2.15x more commits per PR, yet draws 4.52x more review comments, 31% lower acceptance, and 5.16x longer resolution - the verification burden lands on maintainers.
With AI coding tools everywhere, managers and open-source maintainers face a concrete question: can a less experienced developer pumping out code through Copilot, Cursor, or Claude Code stand in for a seasoned one? The optimistic intuition says yes - 92% of developers use AI tools, per the GitHub survey the paper cites, and output visibly speeds up.
Fast output is not the same as usable output. A controlled experiment already threw cold water on the optimism: experienced OSS developers using AI assistants took roughly 19% longer on their tasks, with the bottleneck moving from writing code to verifying it. This paper takes the question to the repository level: group developers by experience and measure the real gap in contribution size, acceptance, review cost, and resolution time for AI-generated PRs. The authors are from LUMS in Pakistan, and the paper appeared at MSR 2026, the mining-software-repositories conference.
The data comes from AIDev, a public corpus of AI-assisted PRs on GitHub covering Copilot, Codex, Claude Code, Cursor, and Devin. The authors take repositories above 100 stars: 33,596 PRs from 1,796 users, filter out usernames containing "bot" or known agent identifiers, and keep 22,953 PRs from 1,719 vibe coders - the human-directs, AI-executes workflow.
Experience follows prior work: lifetime commit count divided by account age, bucketed into quartiles. Top two quartiles (859 coders) form the high-experience group, bottom two (860) the low-experience group. Metrics per PR: commits, files changed, acceptance rate, resolution time in days, and review comment count. Groups are compared with Mann-Whitney U tests (chi-square for acceptance), Benjamini-Hochberg corrected, stratified across 11 PR categories such as bug fix, feature development, and documentation.
| Metric | Low vs high experience | Significance |
| Commits per PR | 2.15x more | p<0.05, 10 of 11 categories |
| Files changed per PR | 1.47x more | p<0.05, 5 of 11 categories |
| PR acceptance rate | 31% lower | p<0.05, 10 of 11 categories |
| Resolution time | 5.16x longer | p<0.05, 10 of 11 categories |
| Review comments | 4.52x more | p<0.05, 6 of 11 categories |
Category extremes: feature PRs average 1.58 commits for the high group versus 4.20 for the low group; style PRs touch 24.29 files versus 70.35; documentation acceptance runs 93.06% versus 75.39%; chore PRs take 0.61 days versus 2.83.
The pattern is consistent: AI helps the low-experience group submit bigger PRs, each unit of output burns multiples of the review budget, and less of it lands. A manual read of the 15 most-reviewed feature PRs from the low group surfaced two recurring frictions. Infrastructure mismatch: the generated code is syntactically fine but blind to CI timeouts and runtime constraints, so the coder commits repeatedly to tune parameters, effectively using CI as a debugger (roboflow/inference PR#1350). Integration friction: the feature logic arrives, but it clashes with the repository's privacy schemas and integration norms, forcing long rework loops (getsentry/sentry PR#94889).
This is the first systematic study crossing AI-assisted contribution with developer experience. The original AIDev paper showed AI-generated PRs get accepted less overall; this one splits the gap by experience, and the numbers are directly usable.
For engineering managers: replacing senior developers with novice vibe coders does not balance on output alone. In this dataset, every unit of writing capacity swapped in brings 4.5x the review comments and 5x the lingering open time. Either review capacity grows with it, or novice training shifts from prompt-writing to verifying what the model produced. For open-source maintainers: within the flood of AI PRs, those from low-experience accounts concentrate the cost, so automated checks and extra-reviewer routing should land on them first.
For vibe coding as a practice, the numbers are a reminder: AI flattened the barrier to writing code, not the barrier to judging whether code belongs in the trunk.
The authors list four: findings depend on how narrowly vibe coding is defined; experience is proxied by commit frequency, conflating activity with skill, so a developer with rich offline experience but a thin GitHub record lands in the low group; per-repository review policies confound acceptance and comment counts, only partly mitigated by category stratification; and despite BH correction, a 10-of-11 significant pattern may still harbor false positives.
Two more from reading it closely. Correlation is not causation: the low group's larger PRs may partly reflect larger assigned tasks, and the AIDev category labels are coarse. And the quartile split averages away within-group variance; on long-tailed metrics like files changed, mean contrasts such as 24.29 versus 70.35 are pulled up by extreme PRs - medians would be more persuasive, and the paper reports only means in the text.