Ortet's Health Index reshuffles open-model rankings across 34 self-hosted models

AI startup Ortet has released its first health-domain benchmark, the Health Index: the team (including keunwoochoi and others) self-hosted all 34 open-weight models (spanning Meta, Kimi, DeepSeek, Qwen, Z.ai, Reflection, and more) on its own NVIDIA Blackwell GPU cluster Orion, evaluating them with 50 sampled tasks across 15 public health benchmarks, and comparing against 8 closed-source frontier models released in September 2026 (GPT-6 Astra/Sol/Luna, Claude Opus 5.5, etc.). The resulting rankings differ markedly from general leaderboards, validating the core claim that "general intelligence rankings are insufficient for assessing health-domain capability."

Confirmed

• Ortet's Health Index covers clinical medicine, biological research, and medical coding, ranking models by equally weighted head-to-head wins across benchmark families.
• Meta's open model Muse Spark Open ranks only 9th on the general AA Index composite, yet tops the health leaderboard among open-weight models; Kimi, DeepSeek, and Qwen follow.
• All 34 open models were self-hosted on Ortet's own NVIDIA Blackwell cluster Orion, without relying on external APIs.
• The comparison set is 8 closed-source frontier models released in September 2026, including GPT-6 Astra/Sol/Luna and Claude Opus 5.5.
• Author keunwoochoi thanked Meta and its chief (the relevant executive acknowledged in the post).

Why it matters

• The benchmark argues that general and domain-specific rankings can diverge sharply: a mid-tier model on composite leaderboards can lead on health tasks, suggesting model selection for medical use shouldn't rely solely on general leaderboards.
• Evaluating all open models self-hosted on uniform hardware reduces bias from differing inference infrastructure, a methodology that is friendly to the open-source community.

2026-10-26 ~ 2026-10-27 · 7 related posts

• Ortet evaluates 34 open-weight models on 50 health tasks, finds general leaderboards insufficient — elmanmansimov · 2026-10-26
• [source] Ortet Health Index: Meta's Muse Spark Open tops open-weight models, Kimi and DeepSeek follow — keunwoochoi · 2026-10-27
• Health-Focused Benchmark Reranks LLMs: Muse Spark Open Jumps from #9 to #1 — keunwoochoi · 2026-10-27
• [source] Open Models Match GPT-6 on MedMCQA but Trail 17 Points on Hard Clinical Cases — keunwoochoi · 2026-10-27
• [source] 34 open models self-hosted on a Blackwell cluster for open health evals — keunwoochoi · 2026-10-27
• Ortet's first health eval: 34 open-weight models, 50 tasks, generic rankings fall short — thegautamkamath · 2026-10-27

1 near-duplicate retellings: kchonyc