FULL STORY

Qwen 3.8 27B: Benchmarks, Backlash and Fixes

Qwen 3.8 27B matched top closed models on benchmarks, then faced overthinking and temperature complaints that users traced to a fixable default setting.

2026-08-17 ~ 2026-08-19 · 3 episodes · 29 posts

Episode 1 · Qwen 3.8 27B "Overthinking" Debate: Tests Point to Necessary Cost for Performance (2026-08-17, 7 posts)

Qwen 3.8 27B sparked "overthinking" criticism after users measured 16K+ default reasoning tokens (@emax relayed the complaints). Around 08-17, multiple users defended the model with local benchmarks, converging on the view that the deep thinking is the necessary cost of approaching much larger models, and users should adjust expectations rather than dismiss the model.

Confirmed

  • @sukazu, after comparing against GLM 5.3 and DeepSeek V4 Flash/Pro on the same tasks, argued the model is not truly overthinking despite using more tokens than 3.6.
  • @maxwell321 ran it locally (3×3090 + Tesla P40, Unsloth UD-Q8KXL quant), finding extensive thinking surfaces fine details from training and brings performance near Claude Sonnet.
  • @AltruisticHeat9531 called 16K+ tokens a necessary sacrifice to rival trillion-parameter models, citing Karpathy's "tokens to think" and SWE-Rebench data.
  • @hiImMate tested thinking levels (UDQ4XL) on an open-ended HTML card task: the xhigh tier consumed 23.8K tokens but outperformed DS V4 Flash, ChatGPT and Claude Opus 5.
  • @ResearchCrafty1804 noted closed-source models (Opus 5, GPT-5.6 Sol) hide internal compute, making raw token comparisons unfair to open models.
  • @wombweed advised disabling thinking or lowering the reasoning budget when speed matters, and trying higher quantization if loops appear; heavy thinking is the product of RL training.
  • @Afinetheorem rated it one of the best local LLMs for high-end consumer GPUs, excellent at coding but among the worst tested at logic and visual reasoning (46/58 vs SOTA).

Why it matters

  • The debate exposes an evaluation asymmetry: open models' reasoning costs are measurable and criticizable, while closed models hide compute server-side.
  • It offers local-deployment users a tradeoff guide: strong value on coding/generation tasks, but logic and vision workloads call for other models.

Episode 2 · Qwen3.8-27B matches top closed-source models on benchmarks while running on a single consumer GPU (2026-08-17, 20 posts)

Artificial Analysis' latest benchmarks show that Alibaba's open-weights Qwen3.8-27B scores 52 on the intelligence index, on par with top closed-source models such as DeepSeek V4 and GPT-5.6 Luna Max, while being runnable locally on a single RTX 3090-class consumer GPU ($2,000-3,000). The community hails this as a 'DeepSeek moment' for local AI and a major milestone for open-source models.

Confirmed

  • Artificial Analysis composite benchmark: Qwen3.8-27B scores 52, comparable to DeepSeek V4 and GPT-5.6 Luna Max, beating all models from four months ago (including Opus); hardware cost $2,000-3,000
  • On the Agentic Index (max reasoning settings), Qwen3.8 Max ties GPT-5.6-Sol at 58 for first place; Qwen3.8 27B scores 51, overtaking GPT-5.6-Terra, GLM 5.2 and DeepSeek V4 Pro 0813
  • Score jumped from 38 in the previous version to 52; the MoE Qwen 3.6 35B A3B scores 32, slightly below the 27B dense model
  • haider1 notes the model is open-weights and its index score of 52 essentially matches GPT-5.6 Luna at max reasoning; udmrzn adds it can run locally on laptops or even in browsers via a custom WebGPU kernel
  • Local tests: remilouf reports 63 tokens/s on Mac Studio; appenz ran complex agentic tasks on a MacBook and saw it beat DeepSeek V4 and Opus 4.7 (roughly 50x larger) on one benchmark
  • WonderRico's local agent-coding benchmark: Medium mode scores above the 3.6 version with far better efficiency (nearly half the requests, one-third fewer generated tokens), reaching DeepSeek v4 Flash 310 level
  • Charts by MikePFrank and enruben plotting Artificial Analysis scores vs. parameter count show the 27B model as a striking outlier; Hesamation places the Qwen 70B-class model at the far left of the size-intelligence Pareto frontier
  • pstAsiatech reports parity with GPT-5.6 Luna and results close to DeepSeek and Zhipu models; users like yacineMTB and ArnoStride call it 'shockingly good' for its size

Unconfirmed

  • A Twitter user relayed by AccBalanced claims the circulating story is actually old news about Qwen 2.5 27B beating SOTA models from six months ago (GPT-4.5, Gemini 2.5 Pro, Claude 4 Opus) with garbled model names/versions; this does not align well with the 08-17/08-18 Artificial Analysis data posts and remains unverified

Why it matters

  • Users (anderspitman, yaboyyoungairvent) argue that frontier-class capability on $2,000-3,000 hardware dramatically lowers the barrier to top-tier AI; community discussions frame it as a 'DeepSeek moment' for local models
  • EAccelerate42 notes its metrics rival trillion-parameter-class models like GLM5.2 (800B) and DeepSeek v4 pro (1.6T); Scobleizer declares the local AI era has arrived
  • Multiple scatter plots confirm the model deviates sharply from the historical size-intelligence trend, sparking debate on architecture and MoE efficiency

Episode 3 · Qwen 3.8 Default Temperature Causes Gibberish, Lower It (2026-08-18, 2 posts)

Users report Qwen 3.8's default temperature of 1.0 causes gibberish and repetition after 10–20K tokens; lowering it to 0.7 reduces verbose thinking and 0.1 largely fixes the degradation.