SWE-bench Science: Can Coding Agents Resolve Engineering Tasks in Science?
Zhipeng Xu, Jiahao Lu, Yining Zheng, Yuxin Wang, Xipeng Qiu
cs.CL, cs.SE
2026-08-20
On 119 scientific software tasks, Claude Code + Opus-5 hits 47.90% Pass@1 vs 96.64% public. Extra science notes lift a weaker model and cut GPT-5.6-sol from 36.26% to 31.87%.
Scientific code is now part of the instrument. A bad patch can corrupt the evidence behind a claim, not only a program output. Most coding-agent leaderboards still collapse that into one pass rate, so a wrong physical object, a symptom-only fix, a missed integration seam, and a principle that will not transfer all look the same. Function-level science coding and paper-reproduction suites exist. A cross-domain, repository-level scientific software benchmark with an explicit failure taxonomy is still thin.
Fudan University and Shanghai Innovation Institute introduce SWE-bench Science: 119 tasks from 98 GitHub repositories across 20 scientific domains, each manually checked for scientific contract, reproducibility, and evaluation validity.
Tasks are split into three paradigms so different skills are not averaged into one number. Issue-driven (52 tasks, 43.7%) starts from a real historical defect and publishes only a coarse phenomenon, with patch location held back. Expert-exploratory (49, 41.2%) hides the root cause and asks the agent to find a mechanism by observation and controlled comparison. Engineering-integration (18, 15.1%) requires wiring a full multi-module capability chain. Chemistry is the largest slice (24), then materials (16), biology (13), biomedical engineering (12), and physics (11); six domains contribute one task each. Non-empty repository lines average 80,600 (174 to 2,029,051). Reference patches add 118 lines and delete 45 on average.
Construction follows a Chain-of-Evidence Protocol: source screening, freeze-and-reproduce the pre-fix snapshot in an isolated container, isolate public materials from hidden oracles, then calibrate hidden validators against boundary cases and hard-coded cheats. The agent sees a repository snapshot, a frozen problem statement, required scientific context, and public tests. Private tests mount only after submission, in a separate evaluator container.
Eight agent configurations are scored, including GPT-5.6-sol with Codex at max, Claude-Opus-5 with Claude Code at max, and DeepSeek-V4-Pro with Claude Code at max. Pass@1 is 1 only when every applicable private test passes. On 91 tasks, scientific auxiliary information (principles, equations, expert diagnoses, upstream-repair notes) can be stripped while the executable engineering context stays fixed.
No configuration wins every column. Claude-Opus-5 posts the best overall Pass@1 at 47.90%, with a 96.64% public score. GPT-5.6-sol leads private score (78.82%), Fail2Pass, and Pass2Pass. DeepSeek-V4-Pro is perfect on public tests and best on Engineering-integration (44.44%). Qwen3.5-397B lands at 14.29% Pass@1.
| Setup | Public | Private | Pass@1 | Issue | Expert | Eng. |
| Claude-Opus-5 + Claude Code | 96.64% | 75.11% | 47.90% | 38.46% | 65.31% | 27.78% |
| GPT-5.6-sol + Codex | 99.16% | 78.82% | 46.22% | 36.54% | 59.18% | 38.89% |
| DeepSeek-V4-Pro + Claude Code | 100% | 73.16% | 42.02% | 26.92% | 57.14% | 44.44% |
| Qwen3.5-397B + Codex | 96.64% | 51.79% | 14.29% | 5.77% | 24.49% | 11.11% |
Frontier models do better on Expert-exploratory than Issue-driven (Opus 65.31% vs 38.46%). Engineering-integration is Opus's weak column at 27.78%. Token plots show Opus taking the top Pass@1 on a moderate budget; GPT-5.6-sol is close with shorter outputs; DeepSeek-V4-Pro spends more input tokens and still trails. Among smaller models, Nex N2 has the better score-per-token tradeoff than DeepSeek-V4-flash or Qwen3.5-397B.
Failures are labeled into four mutually exclusive scientific mechanisms. Opus has the fewest categorized scientific errors (58, plus 4 runtime or evaluation-path failures) and only 2 surface-repair cases. DeepSeek-V4-flash has the fewest generalization misses (6) and the most incomplete-integration failures (48). DeepSeek-V4-Pro has the fewest knowledge or abstraction errors (15).
On the 91-task separable subset, extra scientific text is not uniformly helpful. GPT-5.6-sol moves public/private means from 96.70% / 73.23% to 97.80% / 74.06%, while Pass@1 falls from 36.26% to 31.87% and tokens dip. DeepSeek-V4-flash rises from 16.48% to 23.08% Pass@1, with input tokens jumping from 4.84 million to 7.40 million. Task-level overlap is sharper: GPT solves 8 tasks only with the extra text and 12 only without it; DeepSeek is 9 versus 3. The paper flags these as descriptive deltas with no significance test. Poorly aligned notes can anchor the agent on a supplied story.
Turning public diagnostics green is a different skill from preserving a scientific contract. Near-perfect public scores with sub-50% private exactness is the practical warning. For people building science coding agents, the four failure bins point at missing pieces: object definitions, disciplined exploration, cross-module invariants, and transfer under a changed coordinate system or boundary. Domain write-ups are not free accuracy. Weaker setups absorb hints; stronger ones can be pulled off the executable evidence.
This is an evaluation asset, not a new repair algorithm. Its value is the task design and the failure autopsy, so later work can test, on the same repositories, whether scientific knowledge actually attaches to runnable checks.
The authors say domain counts are still small, so cross-domain ranking is shaky, and the knowledge-use analysis is preliminary. Engineering-integration has 18 tasks; a handful of items can swap 27.78% and 44.44%. Pass@1 is a single attempt, with no pass@k. Failure labels are manual, and one trace can look like both a bad abstraction and a missed neighbor module. The ablation covers two configurations and, by the paper's own wording, is not a causal claim. Six domains with one task each inflate the "20 domains" headline more than they add statistical power. Git history is stripped from snapshots, which blocks issue-thread shortcuts and also makes the setup less like day-to-day scientific maintenance.