Rejected by NeurIPS: study shows a single test-set replica beats uncontaminated irreducible error

RylanSchaeffer · x · 2026-10-15

Rylan Schaeffer's paper "Quantifying the Effect of Test Set Contamination on Generative Evaluations" (arXiv:2601.04301) was rejected from NeurIPS 2026; he disagrees with reviewers and shared it publicly. It's the first systematic study of contamination in generative (vs. discriminative) evals:

Setup: pretrained compute-optimal LMs (34M–344M) on web data mixed with 0–3162 replicas of the MATH test set.

Key findings

Performance rises with contamination and model size, with a sharp jump around 100 replicas; a single replica lets models reach lower loss than the irreducible error of training on the clean corpus—a surprising scaling-law result
Larger models benefit more from any given contamination level
Continued training on fresh data mitigates contamination (dose-response mechanism); SFT on the train set cuts test loss when contamination is low (<10 replicas) but raises it when high (>10)
Solution length and sampling temperature jointly govern memorization decoherence, yielding three regimes: exponentially fast decoherence, brittle memorization, and deterministic lock-in; temperature can shift a model between regimes

Original post →

Rejected by NeurIPS: study shows a single test-set replica beats uncontaminated irreducible error

More from Models