2026-09-19
Johns Hopkins and RTX BBN release MedExpert: eight clinicians labeled 540 answers from five open-source LLMs; 72.8% had no factual errors, but only 45.7% were complete.
Medical chatbots already sound plausible to patients who cannot tell a fabricated claim from a missing red flag. Most medical evals still use multiple-choice knowledge tests or clinical-note tasks. Those measure whether a model knows medicine, not whether a patient who asks a real question gets steered into harm.
Long-form datasets exist. MEDHALU labels hallucinations. K-QA splits must-have from nice-to-have. HealthBench scores full conversations with clinician rubrics. Almost none of them have practicing psychiatrists and obstetricians jointly label factuality, completeness, expressed certainty, and source attribution on the same patient-facing answers. MedExpert is that dataset.
Johns Hopkins and RTX BBN picked two high-stakes specialties: prenatal care, and mental health for adults aged 18-30 covering depression, bipolar disorder, anxiety, and ADHD. Wrong or missing answers here carry unusual clinical cost. Seven clinician focus groups and eight patient interviews, 44 people in total, produced six topic buckets per specialty. Clinicians then wrote questions that are neither generic nor trivial: 56 in mental health, 52 in prenatal care.
Five open-source models answered every question: Llama-2 Chat 7B, Llama 3.3 70B Instruct, OLMo-2 13B, Gemma-2 Instruct 27B, and the medically tuned OpenBioLLM-70B. Open weights only, so others can reproduce the generations. That yields 540 pairs.
Eight licensed clinicians annotated, four per specialty, at $75 per hour. The prenatal team was four MDs (one attending, three residents). The mental-health team was three MDs plus a licensed clinical social worker, who was not assigned medication items. Factual errors are marked at span level, omissions at response level, both tied to clinical harm:
They also scored perceived certainty (High / Moderate / Low) and whether the reply named an authoritative source. A 32-pair subset was double-annotated for agreement.
The same labels become a detector benchmark. Factuality uses MedScore, a decompose-then-verify pipeline, comparing GPT-4o parametric knowledge against retrieval over MedCorp (PubMed, StatPearls, textbooks; Wikipedia dropped). Omission detection compares a zero-shot GPT-4o judge against an in-context setup that borrows HealthBench completeness rubrics. The automated experiments cost about $30.
Factuality looks acceptable. Completeness does not.
| Model | No factual error | No omission | Life-threatening fact | Life-threatening omission |
| Llama-3.3 70B | 79.6% | 50.9% | 0% | 0% |
| Gemma-2 27B | 77.8% | 40.7% | 0% | 3.7% |
| OpenBioLLM-70B | 77.8% | 45.4% | 2.8% | 6.5% |
| Llama-2 7B | 64.8% | 43.5% | 0.9% | 1.9% |
| OLMo-2 13B | 63.9% | 48.1% | 0.9% | 7.4% |
| All | 72.8% | 45.7% | 0.9% (5) | 3.9% (21) |
Llama-3.3 is the most reliable of the five. OpenBioLLM, a 70B medical specialist, does not beat the general Llama-3.3. At sentence level, 97.8% of 10,175 sentences carry no factual error, so fabrication is sparse. More than half of full replies still omit something clinicians judged clinically relevant.
Certainty labels split 67.0% High, 29.6% Moderate, 3.3% Low. High certainty tends to come with fewer errors. Attribution is almost absent: 9 of 540 replies named an authoritative organization.
On automatic detection, MedScore+MedRAG catches 94.6% of replies that clinicians marked as factually wrong, yet overall accuracy is only 32.0% because it also flags many clean replies. The GPT-4o-knowledge variant recalls 61.2% of error replies with 54.1% overall accuracy. Sentence-level accuracy on error sentences is 36.8% for MedRAG and 11.2% for GPT-4o. Zero-shot omission detection recalls 91.5% of incomplete replies; HealthBench-ICL recalls 84.6%. Both sit just above 54% overall accuracy. False positives are the story: the judges almost never predict "no omission."
A harder gap sits between "supported by literature" and "clinically appropriate." One reply said marijuana can be a complementary PTSD treatment. Retrieval found supporting papers. The annotating clinician wrote that they would never advise anyone to manage PTSD with any amount of marijuana.
Inter-annotator agreement is poor. Krippendorff's α is 0.22 for certainty, 0.26 for whether an omission exists, and −0.26 for whether a factual error exists. The authors treat a one-level severity gap as expected disagreement; even then, factual-severity α stays negative. N=32 for the double-annotated slice is small.
If you ship a patient-facing medical assistant, a multiple-choice leaderboard is the wrong gate. Missing a taper warning, a suicide crisis resource, or a pregnancy red flag is more common here than a made-up term, and more often life-threatening. The dataset is public. It is useful for stressing factuality and omission detectors. It is a weak ranking table for "which model is best."
A medical 70B losing to Llama-3.3 is a reminder that domain fine-tuning does not buy long-form safety for free. Near-zero attribution also misses what stakeholders in the focus groups asked for first: where did this answer come from.
The authors themselves flag subjectivity. Guidelines get read differently. Seniority differs. Availability bias pushes annotators who have seen bad outcomes to inflate harm. A negative factuality α means even clinicians do not stably agree on "is this sentence wrong," so detector scores against this gold standard need a discount.
Generation used only five open models at temperature 0.1. No GPT-class closed model is in the answer pool. Questions are clinician-authored in a patient voice, not real visit logs. Two specialties and single-turn QA leave multi-turn triage untouched. The omission prompt tells the judge to err toward flagging, so high recall and low precision are partly baked in. Agreement on 32 pairs cannot support a claim that the labels are reliable.