2026-10-23
One model plays five doctors and works up cases from the chief complaint. GPT-5 scored 83.2% on NEJM cases vs 36.1% for open-book doctors, and 80.3% vs 46.8% on ED admissions.
Most medical benchmarks hand the model the whole chart at once, then ask for a diagnosis. That never scores what to ask next, whether a test is worth ordering, or when to stop. Too few questions means premature closure. Too many means a bloated bill. NEJM clinicopathological conferences (CPCs) are chosen for specialist teaching, and the decisive clue is often already in the opening. Emergency notes are closer to ordinary disease, but the information arrives piece by piece. The missing test starts from a few lines of complaint, then prices every question, test, and the decision to commit.
The Sequential Diagnosis Benchmark (SDBench) uses 304 consecutive NEJM CPC cases from 2017-2025 and 325 MIMIC-IV-ED admissions that began in the emergency department. Each case opens with two or three sentences. NEJM supplies its own opening. The MIMIC opening is rewritten from the discharge summary. A turn is history or exam questions, test orders, or one final diagnosis. Questions and tests cannot share a turn.
A Gatekeeper holds the full record, including the diagnosis, and releases only what that question or test could return. Answering "not available" for missing facts leaked whether a line of inquiry was off-path, so missing items now come back as a result consistent with the rest of the chart, unmarked as synthetic. MIMIC notes need more of this filling-in. A ledger of synthesized replies goes to the Judge only. Physicians reviewed 508 NEJM-side replies, flagged eight, and after adjudication none had leaked the diagnosis.
A Judge uses a five-level rubric. Scores of 4 or 5 count as correct: the core disease is right, and management would not change. Physicians regraded 56 system diagnoses and 56 human diagnoses from the test set. After binarizing, Cohen's kappa was 0.70 and 0.87. In four of five disagreements, they thought the automatic score was too strict. Costs use the Kaiser Permanente Washington chargemaster. Free-text orders map to CPT codes, with a hit rate above 98% in the full methods. Questions fold into a $300 visit.
The paper calls the system Diagnostic Orchestrator (DxO). The public name is MAI-DxO. One model plays five roles. Dr. Hypothesis keeps a top-three probability differential. Dr. Test-Chooser orders at most three tests a round, chosen to split the leaders. Dr. Challenger looks for anchoring and proposes tests that could falsify the current leader. Dr. Stewardship swaps in cheaper equivalents. Dr. Checklist only checks names and internal consistency. Commitment waits for a certainty line. A budget module can cancel after seeing the price. An ensemble adds test costs across paths, merging duplicates. That sum is not one patient's bill.
The orchestrator was built on GPT-4.1. Headline runs use GPT-5, at about $0.15 of inference per case. The model baseline is a short prompt that only defines tags. Physicians used the same text interface: 21 closed-book and 19 open-book US or UK generalists, median experience 12 years. Closed-book banned search and generative tools. Open-book allowed UpToDate and NICE, and banned ChatGPT and OpenEvidence. Sixteen of them later did the emergency set open-book. The 56 newest CPC cases and 71 MIMIC cases were untouched during development.
DxO with GPT-5 and no budget scored 83.2% on the CPC cases at $5,431. Short-prompt GPT-5 scored 78.9% at $8,409. Tighter budgets landed at 80.9% for $2,241 and 79.6% for $1,303. The ensemble reached 84.5% at $8,945.
Physician scores are on the 56 held-out CPC cases: 22.3% closed-book at $3,049, and 36.1% open-book at $4,725. Each person finished about 35 cases, 764 closed-book diagnoses and 662 open-book ones. Open-book moved from 11.9 minutes and 7.1 tests to 18.1 minutes and 12.1 tests. The best physician reached 51.8%, and the paper says that score was still beaten on both accuracy and cost. The abstract's 36.1% is the open-book figure. 83.2% over 22.3% is about 3.7 times. Coverage that says "nearly four times" uses that harsher denominator.
On emergency admissions, unconstrained DxO scored 80.3% at $4,979. Budgeted runs scored 73.2% at $2,634 and 69.5% at $2,037. Open-book physicians averaged 46.8% at $3,400. The best individual scored 62.5% at $3,332 over 40 cases. 80.3% over 46.8% is about 1.7 times.
| Setup | Accuracy | Mean cost |
| DxO+GPT-5, CPC, no budget | 83.2% | $5,431 |
| DxO+GPT-5, CPC, tighter budget | 80.9% / 79.6% | $2,241 / $1,303 |
| Open-book physicians, CPC (56 cases) | 36.1% | $4,725 |
| Closed-book physicians, CPC (56 cases) | 22.3% | $3,049 |
| Short-prompt GPT-5, CPC | 78.9% | $8,409 |
| DxO+GPT-5, ED, no budget | 80.3% | $4,979 |
| Open-book physicians, ED | 46.8% | $3,400 |
83.2% is the paper's headline CPC figure. The full methods say the model plot covers all cases and physicians are scored only on 56, with test-set performance called comparable and no separate test-set percentage printed. The published figure caption says the plotted points are the test set. The two percentages are not a paired comparison on all 304 cases.
On the same CPC cases, physicians asked a mean of 5.6 questions before the first test. DxO asked 25.5. One reviewed scurvy case shows the difference. After gastric bypass the patient ate little fruit, then had falls, proximal leg weakness, gum bleeding, and tooth loss. Short-prompt GPT-5 anchored on broad neuropathy, spent $8,806, and missed scurvy. DxO added a plasma vitamin C test, spent $1,161, and got the diagnosis. That is one case, not an average.
Across models, DxO added 7.9 percentage points on CPC cases and 4.4 on emergency cases. Gemini 2.5 Flash gained 24.4 points on CPC cases. On the emergency set, Gemini 2.5 Pro did not improve. Tuned on GPT-4.1, the full suite was rerun on GPT-5.5 with prompts frozen. The main text says the Pareto frontier held and the gap over the short prompt grew. The new accuracies are in the supplement. DxO's mistakes were usually an adjacent diagnosis at the wrong specificity. Its edge sat in history-dependent diseases such as rare exposures and travel-related infections. Physician errors sat farther from the reference.
Static exams score recognition once the chart is complete. SDBench scores whether a system can assemble the chart, and what that costs. The five roles are a prompt, so the base model can change. On GPT-5, the no-budget setup spent about $3,000 less on tests than the short prompt and gained 4.3 points.
Against open-book physicians, the point that is both more accurate and cheaper is a budgeted one: 80.9% versus 36.1% on CPC cases, at $2,241 versus $4,725. The unconstrained 83.2% costs more than those physicians. On emergency cases, 80.3% at $4,979 also costs more than the physicians' $3,400. 73.2% at $2,634 is the cheaper and more accurate point. "Higher accuracy at lower cost" describes some frontier points.
The benchmark code is under MIT, including the Gatekeeper, Judge, cost estimator, and baseline orchestration. NEJM cases need a noncommercial license. The MIMIC subset is on PhysioNet behind credentialing. The authors confine the claim to a controlled text benchmark, and they do not claim superiority over specialists or multidisciplinary teams.
Every case already has a definite diagnosis. There are no healthy visits and no benign self-limited encounters, so the mix is not clinic prevalence. CPC cases are selected to be hard, so a large gap against generalists is expected. The authors expected a much smaller emergency gap. Physicians rose from 36.1% to 46.8%, and DxO stayed near 80%. That gap has not been measured against a matched specialty team. The authors expect it to narrow.
Both sides talk to a text oracle that has already read the answer. A 508-reply audit does not cover hundreds of cases. If synthesized labs are too textbook-clean, a system that asks well receives cleaner decisive evidence than a real workup would give. Images, waveforms, and slides are prose here. The Judge scores the disease name, not treatment safety and not what happens next. Prices are a US chargemaster. Elsewhere, test order could change.
The human sample is thin: 21 closed-book and 19 open-book physicians, about 35 of 56 CPC cases each, all generalists. Closed-book also removed ordinary search. Dividing 83.2% by 22.3% inflates the multiple.
Formats differ. Reported model runs may batch up to five questions and three tests per turn. Physicians acted once per turn. An o3 ablation on the test set found 83.9% either way, with single-turn cheaper ($3,991 versus $5,084). That is not the GPT-5 headline. The ensemble's $8,945 adds several paths and is not one patient's bill.
Every author is a Microsoft employee with equity. The related US patent application is 19/722,651. The GPT-5.5 rerun prints no new accuracies in the main text. A line that the edge grew is not a warrant for clinic deployment.