DeepSeek benchmark dispute: 128K vs 32K reasoning budget explains conflicting results
danielhanchen · x · 2026-10-31
A public dispute has erupted over Unsloth's third-party comparisons of DeepSeek-V4.1-Flash-1017 (763B parameters, 1M context). The GSQ-RCO team clarified they used standard settings—128K context window and reasoning effort 50—and claims the discrepancy stems from benchmark constraints, not model quality: Unsloth used a strict 32K reasoning budget while their benchmarks allow up to 128K output tokens.
Unsloth's Daniel Hanchen responded that they are rerunning all benchmarks at 128K, with LiveCodeBench and Aider already finished, saying "the results are even more obvious." The shared charts show both quantization stacks performing similarly at 128K, with scores dropping sharply under the 32K budget.
More from Models
- Blogger argues $30K local Kimi K3.1 setup is worth it, despite taking 4.76 years to match GPT-6.1 Terra output — bookwormengr · 2026-10-31
- SGLang boosts DeepSeek V4.1 Flash throughput 67% to 1.09M tok/s per GPU in 4 days — BanghuaZ · 2026-10-31
- DeepSeek ranks near-zero on AA-Omniscience hallucination test, sparking leaderboard credibility fight — teortaxesTex · 2026-10-31
- Dev tasks Claude Fable 5.1, GPT Astra and Grok 4 with overnight long-running job comparison — therealdanvega · 2026-10-31
- DeepSeek V4.1 Flash non-reasoning hits ~1,486 tokens/s on Artificial Analysis at Intelligence Index 18.5 — teortaxesTex · 2026-10-31
- Qwen4 with HC architecture could widen the Q4 vs Q6 quantization gap significantly — Ok-Shower7286 · 2026-10-31