European Open-Source Model SOOFI Accused of Eval Leakage and Overhype

On July 19, researcher @JJitsev posted a series of threads heavily criticizing the European open-source AI project SOOFI. He accused the project of severe evaluation data leakage and misleading marketing, arguing that such selective hype misleads the public and policymakers, ultimately damaging the credibility of the European open-source AI community.

Key Accusations: Eval Leakage and Fabricated Metrics

According to @JJitsev, SOOFI's clone training set had seen most of the evaluation data, with GPQA even containing the test set, making its claim of being a 'frontier-level champion' unconvincing. He noted that if one looks at uncontaminated benchmarks in the report, the performance does not support these strong conclusions. Furthermore, he criticized the project for inventing a 'capability index' that nobody else uses to score itself and claim parity or superiority over the original Nemotron-3-Nano. The project team later responded, stating that the dataset was co-created with TU Darmstadt, they are cleaning up the leakage, and claimed that Nemotron 3 Nano used a smaller eval subset during training.

The 'Sovereignty' Narrative and Project Positioning

@JJitsev argued that SOOFI's announcements failed to clarify that the project merely reproduced Nemotron-3-Nano using NVIDIA's already open-source data and training stack, misleading people into thinking it was entirely independent R&D. He emphasized that retraining an existing open stack on Telekom's B200 machines does not equate to true 'sovereignty.' Real sovereignty requires the ability to understand and improve upon data, architecture, training processes, and scaling laws. Consequently, he summarized SOOFI as an overhyped Nemotron clone.

Spillover Effects and Further Doubts

@JJitsev warned that this exaggerated marketing harms research integrity and creates noise, undermining the long-standing image of German and EU open-source AI as 'transparent and trustworthy science.' He also raised doubts about another European consortium project, EUROPA, suspecting it might follow the exact same route and end up producing yet another 'clone' of Nemotron-3-Ultra.

2026-07-19 ~ 2026-07-19 · 11 related posts

• [source] SOOFI Outlined as a Nemotron Clone Model — JJitsev · 2026-07-19
• SOOFI Called Out as a Mere Nemotron Reproduction — JJitsev · 2026-07-19
• SOOFI's Sovereignty Narrative Slammed as Misleading — JJitsev · 2026-07-19
• SOOFI Accused of Inventing Metrics to Inflate Capabilities — JJitsev · 2026-07-19
• Update on SOOFI Dataset Leakage Controversy — JJitsev · 2026-07-19
• [source] SOOFI Accused of Eval Leakage and Overhype — JJitsev · 2026-07-19
• Questioning Exaggerated Marketing in Open-Source Models — JJitsev · 2026-07-19
• SOOFI Accused of Benchmark Leakage — JJitsev · 2026-07-19
• Against Exaggerating Model Capabilities with Selective Benchmarks — JJitsev · 2026-07-19
• [source] EUROPA Questioned as Just Another Clone Project — JJitsev · 2026-07-19
• Researcher Flags Data Contamination in SOOFI Model Evaluations — JJitsev · 2026-07-19