German Data and Evaluations Skew Comparisons
JJitsev · x · 2026-07-16
This reply corrects a statement in a German model comparison: it should be "better than Nemotron" rather than "versus Nemotron". The author argues that Soofi inherently holds a significant advantage over Nemotron due to having more German training data.
Furthermore, if "rewritten German evaluations" are included in the comparison, using them to prove Soofi's strength over Nemotron becomes invalid. The evaluations themselves are already influenced by training data or language processing methods, distorting the comparative conclusions.
Related event: SOOFI benchmark claims challenged over leakage and baseline reporting(13 posts)→
More from Models
- Anthropic's Terra returns with better writing and 2X usage vs 6.1 Sol — weswinder · 2026-10-27
- User catches model failing to parse 400-paper file, citing stale news sources instead — jungofthewon · 2026-10-27
- Odd eval result: Gemini 4.1 Flash gets more accurate when asked to control its CoT — dfrsrchtwts · 2026-10-27
- Poster surprised that v4.1 flash gets more accurate when asked to control its CoT — dfrsrchtwts · 2026-10-27
- Dev claims Anthropic is baking 'sabotage' into model behavior for 'wrong' engineering — cephaloform · 2026-10-27
- Gemini fails at listing unread emails, tells user to search Gmail manually — gethackteam · 2026-10-27