SWE-Bench Pro Audit Reveals Issues
dl_weekly · x · 2026-07-13
An OpenAI audit discovered that about 30% of the tasks in SWE-Bench Pro are flawed, prompting the withdrawal of its previous recommendation as an alternative to SWE-bench Verified.
This indicates that even highly anticipated benchmarks for software engineering can contain defects severe enough to compromise their reliability. For those using benchmarks for model selection, comparison, or evaluating coding agents, such "benchmark audits" serve as crucial warning signs.
More from coding & agent
- Resin lets an agent build standalone LLM inference engines; vLLM and SGLang optimize DeepSeek kernels differently — AccBalanced · 2026-10-27
- Alchemy adds effect/Workflow support across Durable Objects, Workflows and Lambda Durable Functions — samgoodwin89 · 2026-10-27
- Alchemy's Lambda Web Functions run plain Node web servers on Lambda, no OpenNext needed — samgoodwin89 · 2026-10-27
- Alchemy Domain brings domain-driven design and story testing to Effect backends — samgoodwin89 · 2026-10-27
- Alchemy makes all ~2,850 AWS bindings work from any cloud, not just S3/Lambda/DynamoDB — samgoodwin89 · 2026-10-27
- Production AI agent scored 95% robustness at baseline, 37% under adversarial attack — njyx · 2026-10-27