SWE-bench Audit Reveals Up to 10% Invalid Samples

A recent audit of the SWE Verified benchmark reveals that 5% to 10% of the samples are invalid, introducing significant noise that could affect leaderboard interpretations.

2026-07-25 ~ 2026-07-25 · 2 related posts

• Audit of SWE Verified suggests 5%–10% of samples may still be invalid — xeophon · 2026-07-25
• A repeated SWE Verified critique again points to a 5%–10% invalid-sample rate — xeophon · 2026-07-25