RadLE 2.0: Medical Autonomous Diagnosis Benchmark Released
DrDatta_AIIMS · x · 2026-07-13
This is the main post for RadLE 2.0: a visual reasoning benchmark for autonomous diagnosis in radiology. The goal isn't just to test "how many questions a model can answer correctly," but to evaluate whether models know when to stop and hand over to a human doctor in medical scenarios.
The authors released five core metrics:
- Confidence Weighted: Rewards correct & confident answers, penalizes incorrect & confident ones.
- Reliability: Whether the model is genuinely reliable when providing autonomous-level answers.
- Accuracy: Traditional accuracy rate.
- Safety: Penalizes high-confidence errors.
- Handover Readiness: Whether the model hands over to an expert when uncertain.
The thread also mentions that they included frontier, open-source, and medical VLMs from OpenAI, Anthropic, Meta, Google DeepMind, xAI, NVIDIA, Alibaba Qwen, Mistral, and MiniMax in the leaderboard, but no model achieved the average human expert baseline.
Related event: RadLE 2.0 Released: Benchmarking Medical AI Uncertainty(8 posts)→
More from Models
- Anthropic's Terra returns with better writing and 2X usage vs 6.1 Sol — weswinder · 2026-10-27
- User catches model failing to parse 400-paper file, citing stale news sources instead — jungofthewon · 2026-10-27
- Odd eval result: Gemini 4.1 Flash gets more accurate when asked to control its CoT — dfrsrchtwts · 2026-10-27
- Poster surprised that v4.1 flash gets more accurate when asked to control its CoT — dfrsrchtwts · 2026-10-27
- Dev claims Anthropic is baking 'sabotage' into model behavior for 'wrong' engineering — cephaloform · 2026-10-27
- Gemini fails at listing unread emails, tells user to search Gmail manually — gethackteam · 2026-10-27