RareDiseaseBench: AI agents max out at 43.8% on 35 rare-disease diagnostic workups
kenbwork · x · 2026-11-07
- A new benchmark, RareDiseaseBench, reconstructs 35 diagnostic workups from published rare disease cases. Each task has an agent start from a patient's presentation, decide which tests to order and why, analyze the returned data, and propose a diagnosis and management plan across one or more visits.
- Leaderboard (charts cover pass rate, cost, tokens, wall-clock and tool calls):
- Gemini 4 Argon / PI leads at 43.8% pass rate
- Claude Opus 5.5 / Claude Code and GPT-6 Astra / PI tie at 33.3%
- Most GPT-5.x and Claude Sonnet/Fable models sit in the 10–28% range
- Grok, Gemini Flash, Mistral Large, DeepSeek V4.1 Flash, GLM-5.3, Kimi K3, Nemotron and others all score 0
- The top model is also the most expensive and token-hungry per run — strongest but heaviest.
- Paper and full results are public.
Related event: RareDiseaseBench Tests AI Agents on Rare Disease Diagnosis(3 posts)→
More from Models
- Pre-training ladder trick: test every recipe change on 300M–6B models first — teortaxesTex · 2026-11-07
- 'Hidden GPT-6.2 Astra' claim debunked: fake reveal tricked via one-line console object — teortaxesTex · 2026-11-07
- Inkling 2 Small tops open-weight models on ARC-AGI-2 at $0.27/task, weights coming — LiTianleli · 2026-11-07
- Gemini memory failure reported: model can't recall or write saved memories — Boshirul · 2026-11-07
- Pre-training keeps evolving and will still need humans in the loop for data review — himanshustwts · 2026-11-07
- Golden age of local LLMs: Qwen quants run 131k context on 12GB VRAM — Present_Resource4022 · 2026-11-07