Reproduce it or it doesn't count: the evaluation-side rule that decontamination reports can't replace
NoahPersaud · reddit · 2026-09-20
The author formalizes why training-side decontamination has a hard floor: it's a claim by the party whose score depends on it, over an uninspectable corpus, using matching that misses paraphrases and synthetic derivatives. Commitments and PSI add precision but no third-party verifiability; proof-of-training proposals have been spoofed. The alternative rules out exposure by construction on the evaluation side: no labels, no network, evaluator builds from a named commit and reproduces scores, forward-dated test data where possible. Implemented for small tabular models; the post lists four things reproduction doesn't prove and the open gap of adaptive overfitting via repeated submissions.
More from Safety
- Gemini won't let you keep chat history while opting out of AI training — Google bundles both into one setting — arshdeep · 2026-11-12
- French public WiFi warning: torrent piracy could cost you EUR 7,500 — michalmalewicz · 2026-11-12
- Dan Robinson flags contradiction: China race rhetoric vs loosening GPU export controls — Puzzleheaded-King584 · 2026-11-12
- iKnow open-sources a read-only viewer into macOS's hidden personal knowledge graph — JeremyCMorgan · 2026-11-12
- Debate: chip export controls don't help Nvidia but clearly aid US AI companies — lxrjl · 2026-11-12
- Anthropic Says a Key AI Safety Alarm Has Been Triggered, Days Before Its IPO — Resident-Fall4351 · 2026-11-12