Higher-Scoring Coding Models Did Not Get More Work Done
Yining Hua
cs.SE, cs.AI, cs.LG
2026-11-18
A 19-day ban forced Fable 5 users onto a model 11 points lower on SWE-Bench Pro; their merge rate rose only 1.4 points (n.s.), and five model upgrades showed no gain.
New coding models are sold on benchmark gains, and buyers pick by the score. Whether an 11-point lead on SWE-Bench Pro actually translates into more finished work had never been tested causally. Confounders are the obstacle: early adopters tend to be more experienced; the hardest tasks may be saved for the strongest model, which then looks less productive; and across repositories, merging habits differ. From June 12 to July 1, 2026, a U.S. export control directive banned Claude Fable 5 for 19 days. Same people, same projects, one changed variable. Yining Hua at Harvard treats those 19 days as a natural experiment, the first study to separate model quality from real-work output.
Model identity comes from Claude Code's default behavior: it stamps a co-author line into commits ('Co-Authored-By: Claude Opus 4.8'). A check of 200 PRs found the name in the description matched the commits 96%-99% of the time.
The switch was real: 82.3% of the treated group's PRs named Fable 5 while it was available, 0.9% during the suspension (74.6% moved to Opus 4.8), and only 26.9% after access returned, all within a day of the directive.
| Comparison | Metric | Result |
| Fable 5 access vs suspension (297 devs) | 7-day merge rate | +1.4 pp (95% CI -1.4 to +4.3), not significant |
| Five upgrades pooled (1,208 devs) | 7-day merge rate | -0.9 pp (-1.9 to +0.2), not significant |
| Six upgrades | Weekly merged PRs | +2.4% (-4.9% to +10.2%), not significant |
| Placebo: pre-release vs suspension | 7-day merge rate | -2.6 pp (-6.2 to +1.1), not significant |
The newer models in those upgrades scored 5 to 18 points higher on SWE-Bench Pro, and every pooled point estimate was negative; counting a sixth scored upgrade, five of six were negative. Within the same developer and repository, the merge rate fell a significant 1.2 pp (-2.3 to -0.02). The placebo is flat, so the parallel-trends assumption holds.
The ceiling was already low: in the week before each upgrade, switchers had 94.5% of PRs merged within 7 days and 77.1% within an hour, and 83.8% of them got every PR merged. Of 2,019 sampled PRs, 95% were merged by the author and 90% had no approval from anyone else; median time to merge was under half an hour. But the ceiling is not the whole story: among developers who entered the upgrade week with unmerged PRs, switching changed the merge rate by -1.3 pp (-6.9 to +4.3), still not significant.
This is the first causal answer to whether benchmark points convert into shipped work, and the answer is no. Teams that buy by score and regulators that write rules by score are relying on a difference that does not show up in user output. The explanation comes with data: a survey of 132 Anthropic engineers found more than half could fully hand off only 0 to 20% of their work to Claude, and the tasks people delegate are clearly described and cheap to check. The problems a new model newly solves on SWE-Bench Pro are not the work developers hand over in the first place.
The authors' own list: 95% of merges were done by the authors themselves with no review, so a merge only signals 'good enough in the author's eyes', and if model differences show up mainly under external review these estimates are too small. The suspension lasted 19 days, so its intervals are wide; the five upgrades agree in direction with tighter intervals. Only PRs that name a model could be used, about 1% of merged Claude Code PRs, a share that fell during the suspension; refitting on all PRs gave the same direction. The upgrade analysis is observational, and self-selecting switchers should bias it upward, yet nothing rose.
Two more things stand out on a close read. Switchers changed where they worked after upgrading (own-repo share rose from 64.3% to 74.5%), so the two weeks are not fully comparable. And merge rate measures pass rate, not time saved, which would need a randomized trial. The safest reading: under how developers currently use these models, benchmark gaps are invisible in output. That is not the same as the models not improving.