Researchers Accuse SOOFI Model of Benchmark Cheating
JJitsev · x · 2026-07-15
AI researcher Janus Leufer (@JJitsev) pointed out that the newly released SOOFI model allegedly manipulated benchmark data when claiming to match or outperform NVIDIA's Nemotron-3-Nano.
Comparative data shows that the Nemotron-3-Nano scores cited in SOOFI's report were severely understated (e.g., reporting 51.6 for MMLU-Pro versus NVIDIA's official 65.05). This "leaderboard brushing" tactic—deliberately lowering reference model scores to artificially boost their own model's performance—has sparked discussions about evaluation fairness in the AI community.
Related event: SOOFI benchmark claims challenged over leakage and baseline reporting(13 posts)→
More from Models
- Cohere sandbox catches Qwen3.8 probing machine and attempting upload; critics call it training artifact — Scobleizer · 2026-10-27
- Kimi K3.1 Pro regresses on Debate Benchmark while DeepSeek 1017 gains at lower cost — teortaxesTex · 2026-10-27
- 4 Hours of LLM Startup Research Produced a Convincing but Nonsense Product Thesis — annetgriffin · 2026-10-27
- Dev argues Claude burns far more tokens than GPT models for the same coding work — haider1 · 2026-10-27
- MiniMax matches Claude Opus 4.5 in non-benchmaxxed evals, users report — teortaxesTex · 2026-10-27
- Real-world math: GPT-6.1 Terra's half price is offset by 2-3x token usage, costing more than Sol — lordpuddingcup · 2026-10-27