Audit of SWE Verified suggests 5%–10% of samples may still be invalid

xeophon · x · 2026-07-25

A year-old audit of SWE Verified is used to argue that the benchmark still contains a meaningful number of bad samples. The post says a careful inspection of the top five SWE-bench submissions found remaining problematic cases, including unclear issue descriptions and overly restrictive tests.

The analysis claims:

85 issues were identified that none of the model/scaffold combinations could solve.
40 of the hardest cases were manually reviewed.
14 invalid samples were found in that subset.
Extrapolating from that suggests roughly a 5%–10% invalid-sample rate, or about 6% for the unsolved portion.

The point is that even a widely used coding benchmark can still carry enough noise to affect leaderboard interpretation.

Original post →

Audit of SWE Verified suggests 5%–10% of samples may still be invalid

More from Research