GPQA Benchmark Suspected of Data Contamination
teortaxesTex · x · 2026-07-16
This post highlights a frustrating research phenomenon: someone took the GPQA Diamond evaluation data, made "very minor rewrites," and trained on it for 10 epochs.
The main takeaway is that even light rewrites can severely contaminate benchmark results if the training and evaluation sets overlap. This inflates the model's perceived performance, making the scores unreliable.
Related event: SOOFI benchmark claims challenged over leakage and baseline reporting(13 posts)→
More from Research
- KANLab: an interactive browser playground for Kolmogorov-Arnold Networks — Lukas Niehaus · 2026-10-27
- CMU's LLM2Cert synthesizes stability certificates verified via SMT and Lean — CarnegieMellonU · 2026-10-27
- NVIDIA open-sources FLORA, a robotics data engine ~125x cheaper than GPT-6 Astra — nvidia · 2026-10-27
- RSNA 2026 Deep Learning Labs return: eight hands-on AI classes over four days in Chicago — PRouzrokh · 2026-10-27
- Steering two orthogonal concept neurons makes a model compose "gnome" — Sauers_ · 2026-10-27
- Paper: data schedules and learning rate schedules should be designed together — alifcoder · 2026-10-27