RadLE 2.0: Medical Autonomous Diagnosis Benchmark Released
DrDatta_AIIMS · x · 2026-07-13
This is the main post for RadLE 2.0: a visual reasoning benchmark for autonomous diagnosis in radiology. The goal isn't just to test "how many questions a model can answer correctly," but to evaluate whether models know when to stop and hand over to a human doctor in medical scenarios.
The authors released five core metrics:
- Confidence Weighted: Rewards correct & confident answers, penalizes incorrect & confident ones.
- Reliability: Whether the model is genuinely reliable when providing autonomous-level answers.
- Accuracy: Traditional accuracy rate.
- Safety: Penalizes high-confidence errors.
- Handover Readiness: Whether the model hands over to an expert when uncertain.
The thread also mentions that they included frontier, open-source, and medical VLMs from OpenAI, Anthropic, Meta, Google DeepMind, xAI, NVIDIA, Alibaba Qwen, Mistral, and MiniMax in the leaderboard, but no model achieved the average human expert baseline.
Related event: RadLE 2.0 Released: Benchmarking Medical AI Uncertainty(8 posts)→
More from Models
- NVIDIA says Nemotron 3 Ultra scored 30/42 on the 2026 IMO problems — NVIDIAAI · 2026-07-22
- Gemma-4-26B-a4B reportedly beats Qwen3.6 and Qwen3.5 MoE fine-tunes — JLeonsarmiento · 2026-07-22
- OpenAI is reportedly briefing U.S. lawmakers on its next model family — kimmonismus · 2026-07-22
- Muse Spark 1.1 lands at 1495 on Text Arena with standout agentic-coding price performance — ycombinator · 2026-07-22
- Advanced AI Models Are Becoming Impossible to Plug and Play — emollick · 2026-07-22
- Google Gemini's AI Problem: No Leading Model for Core Workloads — bindureddy · 2026-07-22