arXiv: higher coding benchmark scores don't mean more real work gets merged
rohanpaul_ai · x · 2026-11-23
arXiv paper 2611.23216 by Yining Hua is the first to test whether models scoring higher on coding benchmarks get more real work done.
- Natural experiment: a June 12, 2026 US export control directive banned Claude Fable 5 for 19 days; most users switched to Opus 4.8, 11 points lower on SWE-Bench Pro, with people and projects held constant.
- Difference-in-differences over PRs from 297 Fable 5 users vs 8,908 control developers: Fable 5 access raised the 7-day merge rate by just 1.4 pp [-1.4, +4.3], not significant; weekly merged PR count +2.4% [-4.9%, +10.2%], also not significant.
- Across five 2026 model upgrades (1,208 developers, 15,442 PRs), weighted mean merge-rate change was -0.9 pp, with all five estimates negative.
- Argument: benchmark tasks are mostly already-solved public issues with existing tests, unlike real work with vague specs, project-style constraints and human review; confounders make cross-developer comparisons unreliable.
- Implication: no one knew what an 11-point SWE-Bench Pro gain meant in real work — this paper's evidence says: nothing measurable.
More from coding & agent
- A2A protocol walkthrough: how triage and billing agents collaborate on a refund flow — adnan_hashmi · 2026-11-23
- 开发者用 Claude 35 分钟从 Vercel 迁移到 Cloudflare,月费 70 美元降至 10 美元 — aloncarmel · 2026-11-23
- How A2A protocol works for agent-to-agent delegation in Microsoft Foundry — adnan_hashmi · 2026-11-23
- Aiden Bai demos new dev tool that runs fully locally in the browser — aidenybai · 2026-11-23
- Lantern agent runs on Codex CLI with custom tools in an AWS sandbox — MikePFrank · 2026-11-23
- Lantern: an open-source free-running agent sandbox for Qwen 27B on Codex CLI — MikePFrank · 2026-11-23