Study finds semantic dupes for 78% of CodeForces problems in training data

gleech · x · 2026-09-04

In a thread with Yoav Goldberg on how to measure LLM progress, Gavin Leech cites an arXiv paper on soft contamination: semantic duplicates in training data that n-gram filters miss. Embedding the Olmo3 corpus, authors found duplicates for 78% of CodeForces and exact dupes for 50% of ZebraLogic problems; adding benchmark dupes to training improves scores, and finetuning on dupes also lifts truly held-out scores from the same benchmark. Recent benchmark gains are thus confounded. Proposed fixes: preregistered forecasting-style task streams and a benchmark treadmill with quarterly refreshes.

Related event: Researchers argue benchmarks fall to brute force; real progress should be measured by "discoveries"(6 posts)→

Original post →

More from Research

• Richard Socher: AI could compress a century of scientific breakthroughs into a decade — RichardSocher · 2026-10-21
• Startup Electric Plant builds an AI model to decipher plants' electrical signals — TechCrunch AI · 2026-10-21
• Claude reads the literature to build open human surfaceome atlas, claim by claim — owl_posting · 2026-10-20
• Eval researcher teases guide to subjective-task evals, with a curated resource list — cwolferesearch · 2026-10-20
• IIT Madras article splits LLM memory into five approaches and asks why belief revision is still missing — LimeOptimal539 · 2026-10-20
• Scholar writes AI-assisted book on storytelling values, work nominated for EMNLP best paper — _akpiper · 2026-10-20

Research channel →