Decontamination Reports Can't Save Benchmarks; Evaluators Must Own the Test Set

An evaluation practitioner argues that training-side decontamination claims are inherently unverifiable, spotlighted by OpenAI retiring SWE-bench Verified in February. The fix lies in evaluation-side rules: evaluators must control the test set themselves and require reproducible results.

2026-09-20 ~ 2026-09-20 · 2 related posts

• Reproduce it or it doesn't count: the evaluation-side rule that decontamination reports can't replace — NoahPersaud · 2026-09-20
• Why decontamination reports can't fix benchmark contamination — and what evaluators must do instead — NoahPersaud · 2026-09-20