SFT benchmaxxing backfires: DeepSeek-tuned model gains 172 human rating points
teortaxesTex · x · 2026-10-29
A paper on models finetuning models shows: the DeepSeek-trained model gained 172 points in human ratings but only 4 on benchmarks, while the MiniMax-trained model gained 68 human points while losing 18 on benchmarks. The poster highlights V4.1 Flash as a big L—benchmaxxed with SFT but failed human evaluation—and calls for tests on 1017, K3.1 and more. Takeaway: human feedback gains far exceed benchmark gains; pure benchmaxxing doesn't hold up.
Related event: RSI Arena: AI Agents Self-Train Models with Mixed Results(3 posts)→
More from Models
- Google Announces Gemini 4 Carbon, Its Next Frontier Model, Starting With Cyber Defenders — Snoo26837 · 2026-10-29
- Suhail launches Particle: particle-1 matches performance with ~45% fewer tokens — ycombinator · 2026-10-29
- Defenders of Google's pace point to Genie 4 world model launch and Gemini 4 Argon announcement — haider1 · 2026-10-29
- 'I didn't realize Argon was the mid-sized model': DeepMind naming surprises users — saurabh_shah2 · 2026-10-29
- Unverified 'Gemini 4 Carbon' output showcase thread circulates ahead of any release — cgarciae88 · 2026-10-29
- Rant on confusing model codenames, plus claim that "barium" was shelved for being too spiky — unironictechbro · 2026-10-29