24% of MMLU found verbatim in OLMo-2's training set, Gaperon team shows contamination is systemic
zehavoc · x · 2026-10-13
Gaperon model author zehavoc argues that benchmark evaluation is heavily biased by accidental data contamination, backing it with slides showing: an estimated 24% of MMLU questions appear verbatim in OLMo-2's training set vs 1% for OLMo-1; mainstream models (Llama, Qwen, OLMo, EuroLLM, Gaperon) all score higher on likely-leaked MMLU samples, with OLMo-3 showing a +38.2 gap on Social Sciences; and a Benchmark-in-Haystack experiment showing better data-quality classifiers rank benchmark samples higher, making contamination a side-effect of quality filtering rather than cheating. Oddly, training on test data seems to boost generalization.
Related event: Gaperon team finds 24% of MMLU leaked into OLMo-2 training data(2 posts)→
More from Models
- OpenAI's Dots takes the lead on PersonalAgentBench as task set doubles — adcock_brett · 2026-10-13
- When can local models optimize their own runtime? r/LocalLLaMA debates — Plastic-Stress-6468 · 2026-10-13
- Screenshot shows GPT agent asking to complete CAPTCHAs — are they still useful? — CtrlAltDwayne · 2026-10-13
- OrcaRouter's same-day Qwen3.8-Flash-Next hit piece called out as AI slop on Reddit — buttplugs4life4me · 2026-10-13
- Musk touts Grok Bot as users call it the strongest personal agent yet and push for bundled subscriptions — elonmusk · 2026-10-13
- User says Grok replaced his ChatGPT, Claude and Cursor workflows entirely — elonmusk · 2026-10-13