Stanford's 3-year medical AI benchmark review: GPT-5.5 drops to 28% when tasks must succeed 3 runs in a row

jonc101x · x · 2026-10-28

Stanford HAI published a reflection on three years of building medical AI benchmarks: MedHELM, MedAgentBench, PhysicianBench, and HealthAdminBench. The core finding: a substantial gap between answering questions and completing real clinical workflows.

Key numbers from the thread:

PhysicianBench: GPT-5.5 completed 46.3% of clinical tasks on a single attempt, dropping to 28% when consistent success across three runs is required.
Failure mode: among failures to execute clinical actions, 62% (GPT-5.5) and 73% (Claude Opus 4.6) involved agents correctly documenting what needed to happen in the note, then never placing the order in the EHR.
HealthAdminBench: Claude Opus 4.6 finished only 36.3% of administrative workflows end to end; GPT-5.4 succeeded on 82.8% of individual subtasks but completed just 26.7% of entire workflows.
Clinical decision support and administration/workflow are among the weakest areas — ironically where health systems most need help.
MedHELM is an open, community-owned framework under Apache 2.0.

The author credits Nigam Shah for the article, notes Claude and ChatGPT wrote most of the thread, and points to a Thursday colloquium with Philip Payne (inaugural Chief Health AI Officer, WashU Medicine) titled "Choosing Insight Over Overload in the Age of AI."

Original post →

Stanford's 3-year medical AI benchmark review: GPT-5.5 drops to 28% when tasks must succeed 3 runs in a row

More from coding & agent