Is reasoning_effort a marketing trick? DeepSeek benchmark-tier fight explained
karminski3 · x · 2026-09-15
Blogger karminski3 pushes back on criticism of his DeepSeek benchmark methodology: since other models are tested at max effort, testing DeepSeek alone at high would be unfair — yet commenters insisted he should use high.
His core take: reasoningeffort is a marketing device — it shifts the compute-vs-accuracy tradeoff (and compute billing risk) onto users, while vendors obscure it by claiming benchmarks run at max and pointing to a "sweet spot" (the papers' cost–performance balance) at high, leading some users to believe high is simply best.
- He agrees thinking effort ≠ performance (a post-training failure) but argues the goal should be making them equivalent; DeepSeek v4-1-flash's own tech report calls max "best reserved for the most challenging tasks"
- Quoted @QuantumTransf counters that max often yields only marginal benchmark gains, and 5.6-sol's overthinking at xhigh/max shows excessive effort degrades output quality
The real stakes: how to standardize benchmark tiers, and the business motives behind effort tiers.
Related event: Debate Erupts Over Whether DeepSeek Thinking Intensity Affects Performance(4 posts)→
More from Models
- Liquid AI ships LFM2.5-24B LoRA adapters for Pi and Claude Code, cutting tool calls 40% — yb2698 · 2026-10-31
- 4B model Spark-X2.6 scores 94.6/100 on community benchmark, beating 27B-35B models — mesolimbic-bliss · 2026-10-31
- opencode: $100 of usage gets you DeepSeek V4.1 Flash — intellectronica · 2026-10-31
- Researcher Finds 600+ Bugs in LLM Kernel Code; Claude Blocks Safety Analysis — pmddomingos · 2026-10-31
- Signal65: Snowflake's CoCo agent finishes workflow 4.6x faster than manual — ryanshrout · 2026-10-31
- Mistral Large 4 matches GLM on coding with 47% fewer tokens, halves preview price — petrusenko_max · 2026-10-31