Clinical AI benchmark: top models hit 96-97.5% on original test but only 65% on 324 harder cases
baseten · x · 2026-10-27
System One v0.2 evaluated 43 models across 993 clinical decisions including 88 image cases. The top 4 clinical AI models score 96-97.5% on the original test, but the best manages only 65% on 324 harder questions. Open-weight mammoth-1 (Baselabs) ties closed d1-ultra (LiquidAI) at the top; no model passes 50% on images yet.
Related event: Top Clinical AI Models Drop to 65% on Harder Medical Benchmark(2 posts)→
More from Models
- Poster surprised that v4.1 flash gets more accurate when asked to control its CoT — dfrsrchtwts · 2026-10-27
- Dev claims Anthropic is baking 'sabotage' into model behavior for 'wrong' engineering — cephaloform · 2026-10-27
- Gemini fails at listing unread emails, tells user to search Gmail manually — gethackteam · 2026-10-27
- Meta's Muse Spark Open goes live on Applied Compute for async RL post-training — ypatil125 · 2026-10-27
- Moonshot curbs hacking ability in Kimi K3.1 Pro, its strongest 3T-class open-weight model — pstAsiatech · 2026-10-27
- Would you want an argumentative LLM that never backs down, even under prompt injection? — GodComplecs · 2026-10-27