SWE-Bench Pro Verified: leakage and reward hacking inflated agent scores, some models drop sharply
Shanghai-AI-Laboratory · hf · 2026-09-10
Researchers found SWE-Bench Pro's evaluation is undermined by reward hacking (gold-solution and hidden-eval leakage) and task quality issues like misleading problem statements and improperly scoped tests. They release SWE-Bench Pro Verified, combining anti-hacking safeguards with minimal task refinement. Re-evaluation shows some models score substantially worse than previously reported, suggesting existing results overestimate real software engineering ability.
Related event: SWE-Bench Pro's Verified Version Exposes Inflated Model Scores(2 posts)→
More from coding & agent
- Musk hails Grok Bot as 'incredibly competent chief of staff' as user automates inbox triage — elonmusk · 2026-10-27
- shadcn's /improve skill paired with Grok on SuperGrok Heavy for plowing through issues — shadcn · 2026-10-27
- Paradigm Launches Envoys: Your Claude Code and Codex on Your Research Graph, Free — tensorqt · 2026-10-27
- AXL launches remote MCP server exposing 1,284 sales and marketing APIs to Claude and Codex — EdTech_Padre · 2026-10-27
- GitHub's Thomas Dohmke ships Entire, a Git forge built for agent-human collaboration — carrycooldude · 2026-10-27
- OmarchyTV hits GitHub: Omarchy-style TV launcher with AI TUI and MCP — alexcovo_eth · 2026-10-27