GPT re-analysis shows DeepSeek's agent training gains were understated: +9.7 normalized, +172 human pref
teortaxesTex · x · 2026-10-29
teortaxesTex used GPT to re-check the raw tables of an agent-training comparison paper (2610.33624) and found the headline "only +4 benchmark points" understates DeepSeek's progress:
- DeepSeek: 10 of 16 capability benchmarks improved, mean normalized progress 38.0%→47.7% (+9.7); LiveCodeBench 14.9→26.9, HMMT 3.0→12.1, MMLU-Pro 57.9→62.7; +172 in adjusted human preference vs only +4 on benchmarks — gains come largely from human feedback and style control, not benchmark gaming.
- MiniMax: +68 human rating but -18 on benchmarks (54.5%→43.1%), its final selection missed serious regressions.
- GLM: stronger initial engineering, but weaker adaptation to feedback than DeepSeek.
GPT's verdict: for an established research org with rich context and human direction, DeepSeek's productive adaptation may be the more valuable capability — though DeepSeek used 516 Stage-2 GPU-hours vs GLM's 144, so it's not a compute-matched comparison. It also supports the hypothesis that DeepSeek's model works better inside its own org than unfamiliar environments suggest.
More from Models
- Liquid AI's LFM3: 1.7B tri-hybrid backbone, 237 tok/s on M5 Max, open weights — RobertTLange · 2026-10-29
- Google ships Gemini 4 Carbon two weeks after Argon, tops AA Index at 59 — docmilanfar · 2026-10-29
- Gemini 4 Carbon preview: #1 on AA Index at 59, focused on cyber defense — edchi · 2026-10-29
- Dev Warns Google Carbon Demand Would Collapse Without Argon's $10 Discount Pricing — bindureddy · 2026-10-29
- MiniMax M3.1 Flash scores 43 on Artificial Analysis index as skeptics predict flop — zacharynado · 2026-10-29
- Andon Labs: AI Now Ties Humans on Physical 3D Understanding Benchmarks — ResultBackground2450 · 2026-10-29