Nemotron Comparison Deemed Skewed
JJitsev · x · 2026-07-16
This reply distinguishes between two versions:
- Nemotron 3 Nano: The original version did not use the rewritten benchmark sets for training.
- Nemotron 3 Super: Did use these rewritten benchmark sets.
Therefore, the author argues that the comparison between Soofi and Nemotron 3 is severely compromised, and the results should not be used to draw direct conclusions.
Related event: SOOFI benchmark claims challenged over leakage and baseline reporting(13 posts)→
More from Models
- Anthropic's Terra returns with better writing and 2X usage vs 6.1 Sol — weswinder · 2026-10-27
- User catches model failing to parse 400-paper file, citing stale news sources instead — jungofthewon · 2026-10-27
- Odd eval result: Gemini 4.1 Flash gets more accurate when asked to control its CoT — dfrsrchtwts · 2026-10-27
- Poster surprised that v4.1 flash gets more accurate when asked to control its CoT — dfrsrchtwts · 2026-10-27
- Dev claims Anthropic is baking 'sabotage' into model behavior for 'wrong' engineering — cephaloform · 2026-10-27
- Gemini fails at listing unread emails, tells user to search Gmail manually — gethackteam · 2026-10-27