Stanford HAI medical AI benchmarks show models answer well but fail real clinical workflows

Stanford HAI published a three-year retrospective on its medical AI benchmarks, covering four suites—MedHELM, MedAgentBench, PhysicianBench, and HealthAdminBench. The core finding: a huge gap exists between "answering questions" and "completing real workflows"—chatbots can answer, but as agents executing real clinical tasks they remain far from reliable.

Confirmed

• Benchmark numbers: PhysicianBench measured GPT-5.5 completing clinical tasks at 46.3% on a single attempt, dropping to 28% when required to succeed in three consecutive runs; HealthAdminBench showed Claude Opus 4.6 completing administrative workflows end-to-end at just 36.3%, while GPT-5.4 hit 82.8% on individual subtasks but only 26.7% on full workflows.
• Failure modes: among cases where clinical actions were not executed, 62% for GPT-5.5 and 73% for Claude Opus 4.6 involved the agent correctly recording in the chart what should be done yet never placing the order in the EHR system—"knowing what to do" is not the same as "doing it."
• MedHELM found clinical decision support and administrative/workflow tasks to be among medical AI's weakest areas, precisely where healthcare systems most need AI help; MedHELM is an Apache 2.0-licensed, community-owned open benchmark framework that continues to accept new benchmark contributions.
• Author jonc101x opened the thread with the dichotomy "Chatbots say things. Agents do things." and asked what's still missing before people can trust AI agents to actually perform clinical tasks (rather than just provide information).
• The thread links to the retrospective article and previews Thursday's Stanford Division of Computational Medicine lecture, "Choosing Insight Over Overload in the Age of AI," by Philip Payne of WashU Medicine.
• In the acknowledgments, the author disclosed that the linked article was mostly written by Nigam Shah, while the thread itself was mostly generated by Claude and ChatGPT, then edited by the author with personal commentary added.

Why it matters

• The consistency gap (single-pass vs. three-consecutive-pass success) shows model reliability in medical settings is still insufficient for real clinical tasks, especially the final step of the decision-execution loop—placing EHR orders.
• All benchmarks are open source (Apache 2.0), giving the community a reproducible evaluation standard and pointing to the unresolved core problem of "trusting medical agents."

2026-10-28 ~ 2026-10-28 · 11 related posts

• Chatbots say things, agents do things: a clinician's case for medical AI agents — jonc101x · 2026-10-28
• Knowing what to do is not the same as getting it done in clinical work — jonc101x · 2026-10-28
• [source] Stanford's 3-year medical AI benchmark review: GPT-5.5 drops to 28% when tasks must succeed 3 runs in a row — jonc101x · 2026-10-28
• [source] GPT-5.5 passes 46.3% of clinical tasks once, but only 28% across three runs — jonc101x · 2026-10-28
• Stanford medical AI benchmarks: 62-73% of failures are agents documenting correctly but never placing the EHR order — jonc101x · 2026-10-28
• [source] HealthAdminBench: Claude Opus 4.6 completes just 36.3% of admin workflows end to end — jonc101x · 2026-10-28
• Medical AI is weakest at clinical decision support and admin workflows — exactly where hospitals need it most — jonc101x · 2026-10-28
• MedHELM is an open, community-owned Apache 2.0 medical benchmark framework — jonc101x · 2026-10-28
• What would it take to trust an AI agent to carry out clinical work tasks? — jonc101x · 2026-10-28
• Author credits Claude and ChatGPT for writing most of his Stanford medical AI thread — jonc101x · 2026-10-28
• Stanford colloquium: Philip Payne on "Choosing Insight Over Overload in the Age of AI" — jonc101x · 2026-10-28