Ortet's first health eval: 34 open-weight models, 50 tasks, generic rankings fall short
thegautamkamath · x · 2026-10-27
Ortet released its first health evaluation campaign: 34 open-weight models served and evaluated across 50 tasks and 15 public benchmarks on its Orion NVIDIA Blackwell cluster. Findings: general leaderboards aren't enough for health; open models are competitive on basic medical knowledge and more willing research collaborators; both open and closed models struggle on patient-prediction tasks; more reasoning usually helps but the highest setting doesn't always win; and health rankings can be estimated with a small question subset.
More from Models
- Poster surprised that v4.1 flash gets more accurate when asked to control its CoT — dfrsrchtwts · 2026-10-27
- Dev claims Anthropic is baking 'sabotage' into model behavior for 'wrong' engineering — cephaloform · 2026-10-27
- Gemini fails at listing unread emails, tells user to search Gmail manually — gethackteam · 2026-10-27
- Meta's Muse Spark Open goes live on Applied Compute for async RL post-training — ypatil125 · 2026-10-27
- Moonshot curbs hacking ability in Kimi K3.1 Pro, its strongest 3T-class open-weight model — pstAsiatech · 2026-10-27
- Would you want an argumentative LLM that never backs down, even under prompt injection? — GodComplecs · 2026-10-27