SOOFI Accused of Benchmark Leakage
JJitsev · x · 2026-07-19
SOOFI is accused of eval leakage in its benchmark evaluations. The report allegedly used rewritten evaluation sets—and potentially the test sets themselves—that were already exposed during training, severely undermining its claim of being a "frontier-level champion."
The author points out that looking at LBPP, which was not compromised by the training set, paints a clearer picture: Nemotron 3 Nano scored 38.1, compared to Soofi's 31.0, proving the original model is significantly stronger under un-leaked conditions.
Related event: European Open-Source Model SOOFI Accused of Eval Leakage and Overhype(11 posts)→
More from Models
- GPT-6.1 Terra Now Testable on LMArena in Battle and Agent Modes — arena · 2026-10-27
- OpenAI Launches GPT-6.1 Terra at $1/M Input Tokens, ~9x Cheaper per Task Than Opus 5.5 — VraserX · 2026-10-27
- GPT-6.1 Terra Benchmarks: Sweeps All Six Tests, SWE-bench at 76.2% — mark_k · 2026-10-27
- GPT-6.1 Terra lands: early users praise speed, value and subagent performance — BenBajarin · 2026-10-27
- Leaked GPT-6.1 Terra benchmarks: cheap but 'boring', still far behind Astra and Claude — Angaisb_ · 2026-10-27
- Artificial Analysis Ships Intelligence Index v4.3.2 With GPT-6.1 Terra Comparison — ArtificialAnlys · 2026-10-27