SWE-bench Science Released to Benchmark Coding Agents on Scientific Software Engineering Tasks
geoffwolfe · x · 2026-08-23
SWE-bench Science is a new benchmark for scientific software engineering, comprising 119 tasks across 98 GitHub repositories and 20 domains. It is designed to test the ability of coding agents to resolve engineering tasks in scientific contexts. Evaluations show that even the best-performing agent, Claude Code with Opus-5, achieves a pass@1 score below 50%. The study identifies four recurring failure mechanisms: deficits in scientific knowledge or abstraction, misguided exploration, incomplete repair coverage, and failures to generalize knowledge. Ablation studies reveal that explicit scientific guidance is not uniformly beneficial and requires good alignment to improve performance.
Related event: SWE-bench Science Launches; Top Agents Pass Under 50%(3 posts)→
More from coding & agent
- raylib 6.2 ships with 300+ improvements and 100 new contributors, 20 new API functions — jsuarez · 2026-10-16
- "Vibecoding" is as misleading a label as "AI" and "SI," dev complains — eschadiol · 2026-10-16
- Dev runs Pstack as a Grok bot to audit his own pipeline — it surfaces a pile of hidden bugs — alexcovo_eth · 2026-10-16
- Doc2Harness: New benchmark finds best coding agent passes only 53% across 329 cases — _akhaliq · 2026-10-16
- Perplexity open-sources SPACE, the Firecracker microVM sandbox behind Perplexity Computer — perplexity_ai · 2026-10-16
- Blackstone goes all-in on Factory for autonomous software development — matanSF · 2026-10-16