RadLE 2.0: Medical Autonomous Diagnosis Benchmark Released
DrDatta_AIIMS · x · 2026-07-13
This is the main post for RadLE 2.0: a visual reasoning benchmark for autonomous diagnosis in radiology. The goal isn't just to test "how many questions a model can answer correctly," but to evaluate whether models know when to stop and hand over to a human doctor in medical scenarios.
The authors released five core metrics:
- Confidence Weighted: Rewards correct & confident answers, penalizes incorrect & confident ones.
- Reliability: Whether the model is genuinely reliable when providing autonomous-level answers.
- Accuracy: Traditional accuracy rate.
- Safety: Penalizes high-confidence errors.
- Handover Readiness: Whether the model hands over to an expert when uncertain.
The thread also mentions that they included frontier, open-source, and medical VLMs from OpenAI, Anthropic, Meta, Google DeepMind, xAI, NVIDIA, Alibaba Qwen, Mistral, and MiniMax in the leaderboard, but no model achieved the average human expert baseline.
Related event: RadLE 2.0 Released: Benchmarking Medical AI Uncertainty(8 posts)→
More from Models
- Frontier model weights are near-impossible to steal or self-replicate, argues Bindu Reddy vs AI doomers — bindureddy · 2026-09-11
- Why won't Google open source its STT models while startups ship SOTA open voice models? — techtotechbytechy · 2026-09-11
- User Praises DeepSeek's Model as Surprisingly Fast and Good in Hands-on Test — MaziyarPanahi · 2026-09-11
- Qwen3-8B gets a KV-approximation add-on that halves prefill time without touching the model — teortaxesTex · 2026-09-11
- Pro 20x tier burns 60% of weekly quota in under a day with GPT-6 Astra — rschu · 2026-09-11
- Google isn't honoring its own Gemini Grounded Search pricing: only 289 of 15,000+ requests counted as free — ItalyExpat · 2026-09-11