ERNIE-5.5 caught benchleaking: 96% of HumanEval test set memorized, ranking collapses after decontamination
teortaxesTex · x · 2026-11-16
A new report from GAIR/Xinhu AI, "Benchleaking Is Accelerating," shows benchmark contamination has spread from post-training into pre-training base models. The authors coin benchleaking—test examples entering training data so scores reflect recall, not capability—distinct from benchmaxxing.
Key findings:
- Few-shot contamination probes across 20 benchmarks and 4 public base models show ERNIE-5.5-Flash-Base reproduces 96.3% of HumanEval problems and 85.4% of OpenBookQA questions with full options and reference answers, plus 80.1% of GSM8K and 34.6% of GPQA-Main; MiMo-V2.5-Base leaks 73.8% on code.
- DeepSeek-V4-Flash-Base and Hunyuan Hy3-preview-Base leak almost nothing (0–9%) and score high without memorization.
- The proposed Anti-Hack score zeroes out leaked examples: ERNIE-5.5-Flash-Base drops from 85.2 to 68.5 (-16.6 pts), falling from first to last among the four; MiMo-V2.5 loses 5.0 pts; DeepSeek and Hy3 lose only 1.5 and 1.3 pts.
- Post-training contamination is also worsening: agent traces show explicit hidden-test requirements that persist even when the task is changed to contradict them.
The authors conclude public base models can no longer be assumed clean for downstream research, and call for routine decontamination reporting and contamination-adjusted evaluation.
More from Models
- Screenshot appears to show unreleased "Opus 5.5 Medium" model in Claude Code UI — itsOmSarraf_ · 2026-11-16
- User burns through Grok Code heavy credits in just two days — AIFlow_ML · 2026-11-16
- Sakana AI releases JFinBench: 321 Japanese finance exam questions expose open-model gaps — SakanaAILabs · 2026-11-16
- Dev says GPT-6 Astra is so bad he can't even burn through his Codex Pro 200 quota — bdsqlsz · 2026-11-16
- Pangram's AI-detection bounty goes unclaimed for a week — zero false positives found — ctjlewis · 2026-11-16
- Team surprised its writing model beat Pangram's AI detection without training for it — yawnxyz · 2026-11-16