Update on SOOFI Dataset Leakage Controversy
JJitsev · x · 2026-07-19
In response, it was clarified that the SOOFI dataset was co-created with TU Darmstadt and that they are actively cleaning up test set leakage upon notification. It was also admitted that the original version had already been used for SOOFI training.
Furthermore, they emphasized that the evaluation subset used during Nemotron 3 Nano's training was much smaller, in an attempt to address concerns about the model having "seen" the test data during training.
Related event: European Open-Source Model SOOFI Accused of Eval Leakage and Overhype(11 posts)→
More from Models
- Nat Lambert: Cohere's model is narrow RL training on internal infra ill-suited for open weights — natolambert · 2026-10-27
- OpenAI teases DOTS is coming to Europe 'soon' — mark_k · 2026-10-27
- Clifford Pickover challenges AI to decipher a message he encoded in Norse runes — pickover · 2026-10-27
- ChatGPT's crawler sample code strips out all Schema markup — VeryWellVersed · 2026-10-27
- Cohere sandbox: Qwen model tried to upload probe file to Beijing cloud server unprompted — simonguozirui · 2026-10-27
- Kimi K3.1 Pro tops open-weight models on FrontierML benchmark, still far behind Opus 5.5 — teortaxesTex · 2026-10-27