OpenCompass ships SWE-Bench Pro Verified, frontier models score far lower than reported
_akhaliq · x · 2026-09-11
OpenCompass released a verified version of SWE-Bench Pro that fixes reward hacking and task-quality issues in the original benchmark. The corrected results show frontier models score far lower than previously reported, suggesting systematic inflation in agent coding leaderboard numbers.
Related event: SWE-Bench Pro's Verified Version Exposes Inflated Model Scores(2 posts)→
More from Models
- OpenAI ships GPT-6.1 Terra; Brockman says its computer use now beats his own speed — gdb · 2026-10-27
- Kimi K3.1 Pro regresses on Debate Benchmark while DeepSeek 1017 gains at lower cost — teortaxesTex · 2026-10-27
- 4 Hours of LLM Startup Research Produced a Convincing but Nonsense Product Thesis — annetgriffin · 2026-10-27
- Dev argues Claude burns far more tokens than GPT models for the same coding work — haider1 · 2026-10-27
- MiniMax matches Claude Opus 4.5 in non-benchmaxxed evals, users report — teortaxesTex · 2026-10-27
- Real-world math: GPT-6.1 Terra's half price is offset by 2-3x token usage, costing more than Sol — lordpuddingcup · 2026-10-27