SWE-Bench Pro gets dunked on as a benchmark with 11 repos and 731 tasks
ZainHasan6 · x · 2026-07-22
A joke thread dunking on SWE-Bench Pro, backed by a chart showing how concentrated the benchmark is: across 11 repos and 731 tasks, the top repo accounts for 13.1% of tasks, the top 5 repos for 58.4%, and only 10 repos are effectively represented.
The post frames this as a reason to be far more skeptical of the benchmark’s signal, turning the criticism into a meme about how much less one should “index” on it.
Related event: SWE-Bench Pro Faces Backlash Over Limited Repository Diversity(2 posts)→
More from Fun
- “Tokenization Is Now Free” Turns Into a Classic AI Meme — code_star · 2026-07-22
- Reactions to the AI Security Incident: Open Source, Alignment, and Doomers Weigh In — aiamblichus · 2026-07-22
- Sabine Hossenfelder turns an AI-dooms-math claim into a theoretical-physics joke — KeanuRave100 · 2026-07-22
- Joke imagines “GPT 7 Neptune” as a bank-hacking money machine — dejavucoder · 2026-07-22
- Programming skills are now being used to 100x content creation speed — jackedAJ · 2026-07-22
- ChatGPT turns Amsterdam into a hamster coffee shop after a pun gone wrong — himanshustwts · 2026-07-22