SOOFI Accused of Benchmark Leakage
JJitsev · x · 2026-07-19
SOOFI is accused of eval leakage in its benchmark evaluations. The report allegedly used rewritten evaluation sets—and potentially the test sets themselves—that were already exposed during training, severely undermining its claim of being a "frontier-level champion."
The author points out that looking at LBPP, which was not compromised by the training set, paints a clearer picture: Nemotron 3 Nano scored 38.1, compared to Soofi's 31.0, proving the original model is significantly stronger under un-leaked conditions.
Related event: European Open-Source Model SOOFI Accused of Eval Leakage and Overhype(11 posts)→
More from Models
- Together AI sits down with Meta to explain Muse Spark Open — togethercompute · 2026-10-27
- GPT-6.1 Terra hits ~60% on Agents' Last Exam at $1/task, 9x cheaper than Opus 5.5 — ChrisGPT · 2026-10-27
- ChatGPT keeps telling tech-support users to email 404 Media's Jason Koebler — petrusenko_max · 2026-10-27
- GPT-6.1 Terra called the awkward middle tier, priced at half of GPT-6.1 Sol — cedric_chee · 2026-10-27
- Mystery model "Terra" scores 48 on Artificial Analysis index with no public API — haider1 · 2026-10-27
- GPT-6.1 Terra (Max) Appears on Artificial Analysis Scoring 48 — theanonymousone · 2026-10-27