SWE-Bench Pro gets dunked on as a benchmark with 11 repos and 731 tasks
ZainHasan6 · x · 2026-07-22
A joke thread dunking on SWE-Bench Pro, backed by a chart showing how concentrated the benchmark is: across 11 repos and 731 tasks, the top repo accounts for 13.1% of tasks, the top 5 repos for 58.4%, and only 10 repos are effectively represented.
The post frames this as a reason to be far more skeptical of the benchmark’s signal, turning the criticism into a meme about how much less one should “index” on it.
Related event: SWE-Bench Pro Faces Backlash Over Limited Repository Diversity(2 posts)→
More from Fun
- mark_k: "Eject all doomers from the AI companies — they're destroying you from the inside" — mark_k · 2026-09-11
- rand_longevity: the only thing left to worry about is surviving until aging is solved — rand_longevity · 2026-09-11
- Author retracts 'a16z partner calls for nationalising frontier AI' post: likely a troll — S_OhEigeartaigh · 2026-09-11
- No, Linus Doesn't Code on GitHub — Those Green Squares Are Merge Commits From kernel.org — _jaydeepkarale · 2026-09-11
- Five Years Into the AI Boom, Google Docs Still Red-Underlines 'Compute' as a Noun — ohlennart · 2026-09-11
- Open ECDSA.fail challenge uses AI agents to shrink Shor's-algorithm quantum circuits for Bitcoin keys — StefanoGogioso · 2026-09-11