Overtraining on fresh data dilutes test set contamination, same rejected paper finds
RylanSchaeffer · x · 2026-10-15
Part 3/8 of Schaeffer's contamination-study thread: like discriminative evals, overtraining on fresh data (beyond Chinchilla-optimal) mitigates test set contamination on generative evals.
The authors attribute this to a dose-response mechanism (published separately): overtraining dilutes the proportion of contaminated pretraining tokens. Charts show the crossover point shifting with model size — from 32 test replicas for 34M models down to 1 for 93M, meaning larger models lose their contamination advantage more readily when overtrained.
Related event: NeurIPS Rejects Study Showing Single Contamination Breaks Evaluation Floor(4 posts)→
More from Research
- AlphaNIST: one RL policy solves all 20 NIST assembly tasks with 98.2% success in simulation — hbouammar · 2026-10-15
- Linear Depth Memory: matrix-valued memory tops PreLN and mHC on DepthBench across scales — SonglinYang4 · 2026-10-15
- Neural field planning for hybrid manufacturing cuts estimated build time 68.3% — jiqizhixin · 2026-10-15
- Tev-Gym: 1,170 dense-reward RL environments push long-horizon agentic training — RulinShao · 2026-10-15
- Claude systematically debunks the speculative EDAR-selenium hypothesis about East Asians — teortaxesTex · 2026-10-15
- Microsoft's AutoEnvScaling lets agents build their own RL environments, lifting Qwen3 Terminal-Bench from 41.1 to 53.3 — omarsar0 · 2026-10-15