ERNIE-5.5 caught benchleaking: 96% of HumanEval test set memorized, ranking collapses after decontamination

teortaxesTex · x · 2026-11-16

A new report from GAIR/Xinhu AI, "Benchleaking Is Accelerating," shows benchmark contamination has spread from post-training into pre-training base models. The authors coin benchleaking—test examples entering training data so scores reflect recall, not capability—distinct from benchmaxxing.

Key findings:

The authors conclude public base models can no longer be assumed clean for downstream research, and call for routine decontamination reporting and contamination-adjusted evaluation.

Original post →

More from Models

Models channel →