Harvard paper: higher SWE-Bench scores didn't translate into more merged pull requests
rohanpaul_ai · x · 2026-11-23
A Harvard paper exploits a natural experiment — a US export control directive suspended Claude Fable 5 for 19 days in June 2026, forcing users onto Opus 4.8 (11 points lower on SWE-Bench Pro).
- Tracked GitHub PRs of 297 Fable 5 users against 8,908 developers who stayed on Opus 4.8, using a difference-in-differences model with developer and date fixed effects.
- Access to Fable 5 raised the 7-day PR merge rate by only 1.4 percentage points (not significant); weekly merged PR count also showed no significant gain.
- Repeating the analysis across five 2026 model upgrades (1,208 developers, 15,442 PRs) found no significant merge-rate increases either (weighted mean -0.9 pp).
- Conclusion: coding benchmark improvements don't predict real-world developer productivity, as benchmarks mostly replay already-solved public issues.
More from coding & agent
- Only-Knowing-Code Developers at Risk as the Agentic Software Factory Reshapes Engineering — Pavan_Belagatti · 2026-11-23
- MCP Tool Gateways Should Fail Loudly: A Plausible Answer From a Missing Dependency Is the Real Bug — Emergency-Tutor-7 · 2026-11-23
- Codex Can't Route PCB Traces: Devs Still Lack a Workable AI Tool for Hardware's Last Mile — goldvenetianmask · 2026-11-23
- Uncensored 26.4B model runs on a 12GB Mac at 85 tok/s with 8 parallel agents via Locals — SumitGup · 2026-11-23
- spaCy creator Honnibal shares a cheap-LLM ensemble recipe for building NER data with coding agents — honnibal · 2026-11-23
- 398M tokens, 98% cached: how one dev ran 12.5h of agent game dev on 3% of a $200 Claude plan — Yamapama · 2026-11-23