RareDiseaseBench: 35-case benchmark tests AI agents on rare disease workups
kenbwork · x · 2026-11-07
kenbwork introduces RareDiseaseBench, a benchmark of 35 diagnostic workups reconstructed from published rare disease cases. Rare diseases affect 300 million people worldwide, with diagnosis often taking years.
Design:
- Each agent starts with the patient's presentation and part of the medical record
- Over one or more visits it must decide which tests to request, justify them, analyze results, and propose a diagnosis and management plan
- The multi-turn workup format is closer to real diagnosis than single-shot QA
The author believes concerted AI and robotic automation advances could solve rare disease diagnosis within single-digit years.
Related event: RareDiseaseBench Tests AI Agents on Rare Disease Diagnosis(3 posts)→
More from AGI Musings
- Vitalik Buterin: 'Open weight is not open source' on AI models — wonderwomancode · 2026-11-07
- Analyst predicts China AI consolidation: DeepSeek talent drain leads to 'two big, two small' endgame — teortaxesTex · 2026-11-07
- Mira Murati: Next-gen models never stop listening, chatbots will look like fax machines — Chris_Armstrong · 2026-11-07
- Gary Marcus echoes claim that AI tech hit its limits years ago and will fizzle out — GaryMarcus · 2026-11-07
- Google exec: the race for best AI is irrelevant, scaling affordable access is what matters — docmilanfar · 2026-11-07
- A world without engine noise and human drivers will be more peaceful and safe — davidpattersonx · 2026-11-07