MedExpert dataset: 540 expert-annotated pairs to evaluate medical chatbot reliability
mdredze · x · 2026-09-18
Mark Dredze's group at Johns Hopkins congratulates postdoc Sonal Joshi on completing her postdoc. Their work focused on systems that identify errors and omissions in medical chatbot responses, with one paper published and more on the way.
The published work is MedExpert, a dataset for evaluating medical chatbots, presented at the Fifth Machine Learning for Health Symposium (PMLR 297).
- 540 question-response pairs across two specialties: young adult mental health and prenatal care
- Each response annotated by clinical subject-matter experts for factual accuracy, completeness, and more
- Targets the reliability concerns of patient-facing LLM medical chatbots in clinical settings
- Provides a framework for evaluating automatic error-detection systems in these domains
More from Research
- Base log perplexity is an extremely strong predictor of Epoch and OpenAI ECI, researchers note — stochasticchasm · 2026-11-23
- ECHO-style SFT losses in RL dodge the elicitation-only criticism, researcher argues — stochasticchasm · 2026-11-23
- Idea: humans write high-level proofs, AI fills in lemma details in Lean — theshawwn · 2026-11-23
- Claude points to Lean blueprints, Lamport structured proofs, and Gödel's speedup theorem — theshawwn · 2026-11-23
- Two-person lab GSI claims 250+ breakthroughs across 8 fields with custom AI-built solvers, not brute-force compute — udmrzn · 2026-11-23
- Geoffrey Irving: statistical physics offers a route to obfuscated arguments in scalable oversight — AndrewCritchPhD · 2026-11-23