GPQA Benchmark Suspected of Data Contamination
teortaxesTex · x · 2026-07-16
This post highlights a frustrating research phenomenon: someone took the GPQA Diamond evaluation data, made "very minor rewrites," and trained on it for 10 epochs.
The main takeaway is that even light rewrites can severely contaminate benchmark results if the training and evaluation sets overlap. This inflates the model's perceived performance, making the scores unreliable.
Related event: SOOFI benchmark claims challenged over leakage and baseline reporting(13 posts)→
More from Research
- Causal-only attention for non-generative tasks is wasteful, argues HF engineer — antoine_chaffin · 2026-09-11
- Catholic University of Chile researcher: scaling AI feedback is key to sustainable medical education — julianvarascom · 2026-09-11
- Nature paper images cellular activity across all organs, revealing body-wide circuits — arjunrajlab · 2026-09-11
- SignNet 1M Dataset Released for Sign Language Research — ducha_aiki · 2026-09-11
- ECCV26 Oral: Flow Matching Enables Single-Stage Multi-View Point Cloud Registration — ducha_aiki · 2026-09-11
- InFlux++ Method Released — ducha_aiki · 2026-09-11