Soofi Benchmark Results Suspected of Contamination
JJitsev · x · 2026-07-16
This reply points out that some of Soofi's conclusions might be compromised by contamination from rewritten benchmark sets, making their "superior" comparison results questionable.
The author notes that Tab 5 in Soofi's report includes a benchmark (LBPP) not present in the rewritten evaluation set, where Nemotron 3 Nano leads significantly at 38.1 vs 31.0. This suggests that without using contaminated training data, the results might tell a completely different story.
Related event: SOOFI benchmark claims challenged over leakage and baseline reporting(13 posts)→
More from Models
- Anthropic's Terra returns with better writing and 2X usage vs 6.1 Sol — weswinder · 2026-10-27
- User catches model failing to parse 400-paper file, citing stale news sources instead — jungofthewon · 2026-10-27
- Odd eval result: Gemini 4.1 Flash gets more accurate when asked to control its CoT — dfrsrchtwts · 2026-10-27
- Poster surprised that v4.1 flash gets more accurate when asked to control its CoT — dfrsrchtwts · 2026-10-27
- Dev claims Anthropic is baking 'sabotage' into model behavior for 'wrong' engineering — cephaloform · 2026-10-27
- Gemini fails at listing unread emails, tells user to search Gmail manually — gethackteam · 2026-10-27