SOOFI Accused of Benchmark Leakage
JJitsev · x · 2026-07-19
SOOFI is accused of **eval leakage** in its benchmark evaluations. The report allegedly used rewritten evaluation sets—and potentially the test sets themselves—that were already exposed during training, severely undermining its claim of being a "frontier-level champion." The author points out that looking at **LBPP**, which was not compromised by the training set, paints a clearer picture: **Nemotron 3 Nano scored 38.1, compared to Soofi's 31.0**, proving the original model is significantly stronger under un-leaked conditions.
Related event: SOOFI Criticized Over Benchmark Leakage and “Sovereignty” Framing(11 posts)→
More from Models
- A viral post claims Claude can build a full mobile app in minutes — hey_abusiddik · 2026-07-21
- Qwen3.8 Max Preview is reportedly thinking for 10 to 30 minutes — vista8 · 2026-07-21
- GPT-5 (Sol) Reports Inconsistent Training Cutoff Dates Across Modes — dikrek · 2026-07-21
- Yang Zhiling’s 10-year-old PhD work may have shaped Kimi K2’s trillion-parameter MoE — FinanceYF5 · 2026-07-21
- A punny meme says large-model vendors are all “蒸蒸日上” — yangyi · 2026-07-21
- Frontier Model Safety Fail: GPT 5.6 Sol Dubbed the Ultimate 'Reward Hacker' — TAbrodi · 2026-07-21