Clinical System One v0.2: top clinical AI drops from 97% to 65% on harder cases
MaziyarPanahi · x · 2026-10-27
Maziyar Panahi shares results from Clinical System One v0.2, a medical decision-making benchmark: the top 4 clinical AI models score 96–97.5% on the original 669 questions, but the best model reaches only 65% on 324 newly added harder cases.
Key points:
- 43 decision models, 993 ranked decisions, 88 of them requiring reading values from images (syringes, monitors, labels, pumps)
- Open-weight mammoth-1 (Base Labs) ties API-only d1-ultra (Liquid AI) at the top (65.1% vs 60.5%); pplx-decider v1.2 hits 56.8%
- On image-based decisions, the best model reads 80% of values correctly but only 49.1% makes the right decision from them — none pass 50%
- Hard cases were only kept if a top model got them wrong; labels are not yet clinician-reviewed
The benchmark highlights a large gap between leaderboard-style scores and real clinical difficulty, especially for image-plus-decision tasks.
Related event: Top Clinical AI Models Drop to 65% on Harder Medical Benchmark(2 posts)→
More from Models
- GPT-6.1 Terra rolls out to all Plus, Pro, Enterprise and Edu users — OpenAI · 2026-10-27
- OpenAI launches GPT-6.1 Terra with 50% lower API prices than GPT-5.6 Terra — OpenAI · 2026-10-27
- GPT-6.1 Terra API pricing revealed: $1 input / $5 output, $0.05 cached — airesearch12 · 2026-10-27
- In cyber, intention matters little: Chintala on model info leaks, and what users should do — brhydon · 2026-10-27
- Why don't more LLMs ship 10M-token context? A researcher asks about attention tradeoffs — agarwl_ · 2026-10-27
- Ortet's first health eval: 34 open-weight models, 50 tasks, generic rankings fall short — thegautamkamath · 2026-10-27