Counterintuitive: capping Qwen3.8-27B's thinking budget lifts GPQA score from 115 to 160
AIFlow_ML · x · 2026-11-18
A chart circulating from @niklaslenzai: same Qwen3.8-27B on TensorFold, GPQA Diamond (out of 198), with the only variable being a --thinking-budget cap:
- Mac Studio M5 Ultra (MLX 4-bit, cap 4096): 115 → 160
- 1× DGX Spark (NVFP4, cap 8192): 131 → 172
- RTX 5090 (NVFP4, cap 8192): 131 → 172
Capping thinking didn't hurt accuracy — it improved it substantially, consistently across three different hardware setups (measured by the source, not re-run by the reposter). Takeaway: longer thinking isn't always better; worth trying a thinking budget on local deployments.
More from Models
- Tencent open-weights Hunyuan Hy4 flash: 224B MoE with 12B active params and 1M context — TencentHunyuan · 2026-11-18
- Anthropic's own summarizer refuses its requests, exposing layers of model guardrails — zetalyrae · 2026-11-18
- PenaltyBench: Gemini 4 Carbon beats VAR referees at calling penalties, 63.3 vs 62.3 — ziqiao_ma · 2026-11-18
- Claude models 'discover' body parts only when touched, says researcher — repligate · 2026-11-18
- GPT-6.2 Sol hands-on: Fast mode finishes tasks noticeably quicker, says KOL — op7418 · 2026-11-18
- Dev: I trust Discord vibe checks over benchmarks now — and that's terrible science — burny_tech · 2026-11-18