SWE-Bench Pro Faces Backlash Over Limited Repository Diversity

The AI benchmark SWE-Bench Pro is facing criticism for its lack of representativeness, as users point out it only contains 11 repositories and 731 tasks. Commenters argue this limited diversity makes it an unreliable metric for evaluating true coding capabilities.

2026-07-22 ~ 2026-07-22 · 2 related posts

• SWE-Bench Pro has only 11 repos, and that may be too narrow for real SWE work — ZainHasan6 · 2026-07-22
• SWE-Bench Pro gets dunked on as a benchmark with 11 repos and 731 tasks — ZainHasan6 · 2026-07-22