DeepSeek open model changed answers on 51 of 1,273 medical questions — thinking 12x longer and half wrong
MaziyarPanahi · x · 2026-10-26
Maziyar Panahi ran the same 1,273-question USMLE-style exam through DeepSeek's open model (V4.1-Flash-1017, MIT, reasoning on) three times at temperature 0. On the 1,222 questions where answers stayed consistent, median thinking was 308 tokens with 96.5% accuracy. On the 51 questions where the answer changed between runs, the model thought 12x longer (median 3,867 tokens) and got only 47.7% right — suggesting excessive thinking time is a cheap proxy for model uncertainty.
More from Models
- Running a 125B model on 8GB VRAM with Strata: 900 t/s prompt processing, 30 t/s generation — BleynSpecnaz · 2026-10-26
- More OG Hugging Face profiles to follow: local AI, diffusion, kernels and transformer.js — ariG23498 · 2026-10-26
- 9 Hugging Face profiles worth following, from TRL agents to vLLM kernels — ariG23498 · 2026-10-26
- Terence Tao says OpenAI went 'code red' to beat Anthropic on Navier-Stokes, pausing $200 Pro signups — Hesamation · 2026-10-26
- LoopUno: 52-Minute LoRA Training Makes 3B LLM Draft 2 Tokens per Step Losslessly — LucaAmb · 2026-10-26
- OpenAI enterprise price cuts spark debate as users cite quota frustration and churn — teortaxesTex · 2026-10-26