Gaperon team finds 24% of MMLU leaked into OLMo-2 training data

The Gaperon team revealed that benchmark contamination is distorting evaluations, with 24% of MMLU test questions found in OLMo-2's training data. Surprisingly, training on test data may even improve out-of-domain generalization, though it makes results harder to interpret.

2026-10-13 ~ 2026-10-13 · 2 related posts

• 24% of MMLU found verbatim in OLMo-2's training set, Gaperon team shows contamination is systemic — zehavoc · 2026-10-13
• Gaperon team: benchmark contamination skews evals, yet test-data training may aid generalization — antoine_chaffin · 2026-10-13