Test set contamination study rejected by NeurIPS: single exposure beats irreducible error floor
RylanSchaeffer · x · 2026-10-15
Researcher Rylan Schaeffer shared his NeurIPS 2026-rejected paper Test Set Contamination of Generative Evaluations, publishing it anyway after disagreeing with reviewers.
Key findings on how contamination differs between generative and discriminative evals: even a single exposure to the test set during pretraining yields loss lower than the irreducible error of an uncontaminated corpus — effectively infinite CE gain in compute-efficiency terms. The attached chart shows MATH test loss dropping steadily as replica count grows from 1 to 316.
The thread also covers a dose-response mechanism for how fresh-data overtraining mitigates contamination.
Related event: NeurIPS Rejects Study Showing Single Contamination Breaks Evaluation Floor(4 posts)→
More from Models
- Google can't rule out Loss-of-Control threshold for Gemini 4 Argon, table shows — TheZvi · 2026-10-15
- Indra OCR Fact-Check: Accuracy Claims Hold, But the 17x Speed Claim Is Really ~3x — TheOyinbooke · 2026-10-15
- Frontier AI duopoly cracking as Google and Meta ship new models — danshipper · 2026-10-15
- GPT-6 Astra solves 49 of 50 unpublished physics problems at 98%, catching physicists' errors — 141_1337 · 2026-10-15
- Rejected by NeurIPS: study shows a single test-set replica beats uncontaminated irreducible error — RylanSchaeffer · 2026-10-15
- New SWE-bench Multimodal leaderboard: Claude Opus 5.5 leads at 70.21% on visual bugs — OfirPress · 2026-10-15