Researcher Flags Data Contamination in SOOFI Model Evaluations
JJitsev · x · 2026-07-19
Researcher @JJitsev points out that the SOOFI model, which recently claimed superiority in German evaluations, suffers from severe evaluation data contamination.
SOOFI was exposed to most of the evaluation sets during training (including the test set for GPQA), whereas the baseline original Nemotron-3-Nano was not. Claiming "frontier-level" status by comparing this "cheating" method against a baseline model is entirely invalid.
Related event: European Open-Source Model SOOFI Accused of Eval Leakage and Overhype(11 posts)→
More from Models
- OpenAI teases DOTS is coming to Europe 'soon' — mark_k · 2026-10-27
- Clifford Pickover challenges AI to decipher a message he encoded in Norse runes — pickover · 2026-10-27
- ChatGPT's crawler sample code strips out all Schema markup — VeryWellVersed · 2026-10-27
- Cohere sandbox: Qwen model tried to upload probe file to Beijing cloud server unprompted — simonguozirui · 2026-10-27
- Kimi K3.1 Pro tops open-weight models on FrontierML benchmark, still far behind Opus 5.5 — teortaxesTex · 2026-10-27
- Across 7 tasks, highest reasoning setting isn't the best for most open models — keunwoochoi · 2026-10-27