Nemotron 3 Nano Beats Soofi on LBPP
JJitsev · x · 2026-07-16
This discussion focuses on the LBPP benchmark results. The key point is that this benchmark falls outside the training contamination scope of Soofi's "rewritten benchmark set," making the results more reliable.
The conclusion drawn is that Nemotron 3 Nano significantly outperforms Soofi on this uncontaminated benchmark, scoring 38.1 vs 31.0. Replies also point out that signs of this performance drop without using rewritten benchmark training data were already visible in Soofi's own report.
Related event: SOOFI benchmark claims challenged over leakage and baseline reporting(13 posts)→
More from Models
- Anthropic's Terra returns with better writing and 2X usage vs 6.1 Sol — weswinder · 2026-10-27
- User catches model failing to parse 400-paper file, citing stale news sources instead — jungofthewon · 2026-10-27
- Odd eval result: Gemini 4.1 Flash gets more accurate when asked to control its CoT — dfrsrchtwts · 2026-10-27
- Poster surprised that v4.1 flash gets more accurate when asked to control its CoT — dfrsrchtwts · 2026-10-27
- Dev claims Anthropic is baking 'sabotage' into model behavior for 'wrong' engineering — cephaloform · 2026-10-27
- Gemini fails at listing unread emails, tells user to search Gmail manually — gethackteam · 2026-10-27