Ortet evaluates 34 open-weight models on 50 health tasks, finds general leaderboards insufficient
elmanmansimov · x · 2026-10-26
AI startup Ortet released its first health evaluation: 34 recent open-weight models self-hosted on its Orion NVIDIA Blackwell GPU cluster, tested on 50 tasks from 15 public health benchmarks, plus 8 closed frontier models from September 2026 for comparison. About 75% of the open models were released in the prior six months.
Key findings:
- General leaderboards are not enough for health-specific capability
- Open models are competitive on basic medical knowledge and more willing research collaborators
- Both open and closed models struggle on patient-prediction tasks
- More reasoning usually helps, but the highest setting doesn't always yield the best observed score
- Health rankings can be estimated with a small subset of questions
This is the first step toward Ortet's goal of continuously improving intelligence across clinical, biological, and operational health, with open weights enabling a fully controlled eval-train-reeval loop.
More from Research
- Open Models Match GPT-6 on MedMCQA but Trail 17 Points on Hard Clinical Cases — keunwoochoi · 2026-10-27
- Health-Focused Benchmark Reranks LLMs: Muse Spark Open Jumps from #9 to #1 — keunwoochoi · 2026-10-27
- Open-source Latch compiles PyTorch into training megakernels, 2.75x faster on H100 — sguada · 2026-10-27
- Goodfire commits $10M to a 'Human Genome Project for AI' to reverse-engineer frontier models — CatAstro_Piyush · 2026-10-27
- π0.5 study: raising demos from 10 to 438 lifts Place Ball in Bowl success from 7% to 83% — stingning · 2026-10-27
- HandsOn benchmark: 66 verified tasks and 18,000 human demos for dexterous manipulation — stingning · 2026-10-27