Gaperon team: benchmark contamination skews evals, yet test-data training may aid generalization
antoine_chaffin · x · 2026-10-13
In a discussion on benchmark contamination, Gaperon team member antoinechaffin argues that contaminated data can yield out-of-domain gains, but this makes results hard to read and in-domain gains should be tagged as such — while staying OOD forever isn't a solution either. Quoting the original post: contamination biases the entire evaluation process, yet training on test data seems to increase models' generalization abilities, a point they made when releasing the Gaperon models.
Related event: Gaperon team finds 24% of MMLU leaked into OLMo-2 training data(2 posts)→
More from Models
- Fine-tuning speech models for Algerian Darija: a team's four open questions — Fair_Bed_914 · 2026-10-13
- Google owns every dimension of personal context yet ships no unifying AI product — signulll · 2026-10-13
- Jason: 99% of enterprises don't need esoteric math prizes, open source models will win — markjeffrey · 2026-10-13
- Qwen3.8 Flash Next IQ3_S hits 90 tk/s locally, low-poly 3D generation beats expectations — migsperez · 2026-10-13
- Meta ships muse-wonder-full-1.0 to UK same-day as US, researcher says thanks — SamuelAlbanie · 2026-10-13
- SubQ launches Snap, a 384K-token decision model with free output at $0.05/M input tokens — heyshrutimishra · 2026-10-13