12 open models all ace a medical exam; Kimi K3 leads, Qwen3.8 27B hits 94.5% on one GPU
MaziyarPanahi · x · 2026-11-03
The author ran a medical exam through 12 of the strongest open models, three runs each: all scored between 92.8% and 96.4%, with Kimi K3 leading and the single-GPU-sized Qwen3.8 27B hitting 94.5%.
One contentious case stood out: a 32-week pregnant woman with ruptured waters and contractions, where the answer key says cesarean section — but all 36 model answers said induce labor. 1,013 questions were answered correctly by every model in every run. The author is asking physicians which answer is right, highlighting a gap between model judgment and the official key.
Related event: 12 Open-Source Models Take a Medical Exam, All Challenging the Answer Key(2 posts)→
More from Models
- New Kaggle benchmark transcribes 1621–1737 Portuguese records; Gemini 4 Argon leads at 0.86 — mariofilhoml · 2026-11-03
- Yoav Goldberg speculates RL training included tasks on interpreting and editing model weights — yoavgo · 2026-11-03
- Baseten launches SubQ Snap 1.1: a 384K-context 'System 1' decision model tied for #1 on Decision Index — baseten · 2026-11-03
- Open-weights licenses tighten: Qwen, Kimi, GLM and Step add provider conditions, DeepSeek stays pure MIT — teortaxesTex · 2026-11-03
- ChefNMR2 benchmark: 82.3% top-1 accuracy on 2,000+ molecules vs 52% for best LLM — ZhongingAlong · 2026-11-03
- User asks Kimi K3 to open its weights, self-fix benchmarks, and score low on an 'evil eval' — eigenron · 2026-11-03