Synthetic Hospital: An Open, Verifiable, Physician-Validated Longitudinal EHR Benchmark
Christine Park, Valerie Chen, Tim Dettmers
cs.AI, cs.DB
2026-09-25
CMU builds 1,268 synthetic patients and 5,602 encounters. Physicians spot fakes at 53%; best model hits 0.73 F1 on problem lists and misses about half the key findings.
Medical language models already look strong on multiple-choice exams and still sit outside clinical workflows. Real work means reading a chart that spans months or years: sparse notes, errors, evidence split across progress notes, labs and imaging. Public benchmarks are mostly single vignettes or exam items. Real EHR corpora such as MIMIC cannot be openly shared, and their labels are whatever was documented. When a model fails, it is hard to tell whether the reasoning was wrong or the chart was incomplete.
Christine Park, Valerie Chen and Tim Dettmers at CMU built a fully synthetic, openly redistributable hospital in which every fact has a provenance chain back to public medical-education material.
The pipeline builds the knowledge graph before any narrative. Board-style questions become encounter candidates. Kimi 2.5 extracts primary, differential and secondary diagnoses plus typed findings; codes are then grounded deterministically to ICD-10-CM, SNOMED CT and LOINC, with LLM codes treated only as candidates. Every node keeps its source id. Longitudinal patients are assembled by graph clustering: demographically compatible, and each added encounter must share a correct or secondary diagnosis with the cluster. Each source question belongs to one patient. That yields 1,268 patients from 5,602 source questions. Kimi 2.5 then writes a profile and limited cross-encounter HPI; problem and medication lists propagate by rule. Almost every label is a deterministic function of the graph. Imaging indication is the exception: a short order and reference question are generated, then scored by concept overlap rather than wording.
Records are served through FHIR R4, role-based access, and an Epic-style 13-tool function-calling interface. Four tasks, five scored variants: longitudinal problem list (severity-weighted F1), whole-chart summarization (finding-level F1), specialty-conditioned summarization, evidence retrieval (P@5 / NDCG@10), imaging indication (concept F1). Splits are at the patient level: 200 public, 800 train with full labels (scores in 0 to 1, usable as verifiable rewards), 268 held-out scored only through the official scorer. Source questions do not cross splits. Diagnoses may recur: 54% of held-out diagnoses also appear in training, 46% do not.
Ten licensed physicians reviewed ten charts (five real MIMIC-IV records converted into the same schema by rules, five synthetic, matched on domain, sex, age and length). Across 100 judgments, accuracy at real-versus-synthetic was 53% (95% CI 43-63%, binomial p=0.62), indistinguishable from chance. The graph recovered 111 of 119 physician-preregistered diagnosis relationships (93%).
None of ten models approaches ceiling on the public split:
| Model | Dx F1 | Summ. F1 | Retrieval P@5 | Imaging F1 |
| Gemini 3.1 | 0.681 | 0.502 | 0.809 | 0.496 |
| GPT 5.3 | 0.703 | 0.489 | 0.833 | 0.518 |
| Kimi 2.5-thinking | 0.732 | 0.532 | 0.811 | 0.517 |
| Opus 4.6 | 0.615 | 0.550 | 0.816 | 0.473 |
| Gemma 3 27B | 0.287 | 0.385 | 0.804 | 0.418 |
| Physician mean (7 on 13 patients) | 0.664 | 0.051 | 0.888 | 0.282 |
| Physician range | 0.21-0.89 | 0.01-0.10 | 0.79-0.96 | 0.21-0.35 |
Best longitudinal diagnosis is Kimi 2.5-thinking at 0.73, level with the physician mean and below the best physician at 0.89. Best summarization is still 0.55 (Opus 4.6), missing about half of graph-defined key findings. Physicians score 0.05 on the same finding-level F1; that number says more about a fragmented metric than about clinical writing. Retrieval is the mature task: every model clears P@5 of 0.74, and the physician mean of 0.89 is still higher.
A multi-turn agent loop hurts when the relevant chart is already in context, with drops of 0.07 to 0.19 on summarization and retrieval. Longitudinal diagnosis is the exception, because evidence has to be assembled: self-retrieving agents gain 0.14 to 0.34, and Llama 4 Scout moves from 0.554 single-turn to 0.894. Failed sessions consume about 145k input tokens on average, versus 60k when they finish.
Re-rendering 100 patients with GPT 5.3, labels held fixed, does not move Kimi's margin with the generator. Mean absolute change on primary metrics is 0.034.
Few clinical AI benchmarks are open, labeled from a constructed patient state, and aimed at longitudinal chart work. The training split has deterministic scores, so it can feed reinforcement learning with verifiable rewards. For people building clinical agents the message is concrete: retrieval is already usable; longitudinal synthesis and summarization are not; if the evidence is already in context, an extra agent loop mostly burns tokens.
The physician Turing test uses ten charts and tests record-level plausibility, not population epidemiology. The case mix is education-derived, so rare and atypical presentations are thin. Missingness, contradictions and documentation errors are not simulated. The agent study covers three models, 100 patients and one scaffold. Diagnoses written in the chart but absent from the graph get no extra reward; chart-neutral scoring at least does not penalize them. Kimi both helps build the graph and sits on the leaderboard. A generator-swap check blocks a style-familiarity story; the original graph still comes from the same model.