SWE-bench Science Released to Benchmark Coding Agents on Scientific Software Engineering Tasks
geoffwolfe · x · 2026-08-23
SWE-bench Science is a new benchmark for scientific software engineering, comprising 119 tasks across 98 GitHub repositories and 20 domains. It is designed to test the ability of coding agents to resolve engineering tasks in scientific contexts. Evaluations show that even the best-performing agent, Claude Code with Opus-5, achieves a pass@1 score below 50%. The study identifies four recurring failure mechanisms: deficits in scientific knowledge or abstraction, misguided exploration, incomplete repair coverage, and failures to generalize knowledge. Ablation studies reveal that explicit scientific guidance is not uniformly beneficial and requires good alignment to improve performance.
Related event: SWE-bench Science Launches; Top Agents Pass Under 50%(3 posts)→
More from coding & agent
- Anthropic GA's Computer Use and Skills API for agents to operate software and apply expertise — krishnan · 2026-08-23
- ThoughtDAG: Local-first infinite canvas for explicit LLM context management — Lopsided_Scarcity979 · 2026-08-23
- Reverse-engineering Claude Code agents: CLI tool enables cross-agent communication — KirkNewcombe · 2026-08-23
- Help: Creating a face count detection loop in ComfyUI workflow — Scardra · 2026-08-23
- Mercury Open Sources 132 AI Agent Skills for Coding Workflows — tom_doerr · 2026-08-23
- Tested ChatGPT plugin summarizing 40 tabs, turning it into a multi-session orchestrator — koltregaskes · 2026-08-23