NeurIPS Rejects Study Showing Single Contamination Breaks Evaluation Floor

Stanford researcher Rylan Schaeffer revealed a NeurIPS-rejected study showing a single contaminated copy can push model performance below irreducible error, adding that overtraining on fresh data mitigates the effect and that a Math Verify Score bug in LM Evaluation Harness was fixed.

2026-10-15 ~ 2026-10-15 · 4 related posts

• Test set contamination study rejected by NeurIPS: single exposure beats irreducible error floor — RylanSchaeffer · 2026-10-15
• Overtraining on fresh data dilutes test set contamination, same rejected paper finds — RylanSchaeffer · 2026-10-15
• Rejected by NeurIPS: study shows a single test-set replica beats uncontaminated irreducible error — RylanSchaeffer · 2026-10-15
• Math Verify Score bug fixed in LM Evaluation Harness; pre-2026 paper scores may need reruns — RylanSchaeffer · 2026-10-15