RareBench eval: Claude Fable 5.1 tops rare-disease diagnosis while Nemotron 3 Ultra scores 0%
danielmckinn0n · x · 2026-09-05
The author introduces RareBench, an independent benchmark testing whether models can diagnose a child's rare disease from genome and phenotype alone — a benchmark labs presumably aren't benchmaxxing (yet).
Key results:
- Anthropic's Claude Fable 5.1 remains on top, narrowly edging out Fable 5.0.
- Google's Gemini 3.8 regressed versus 3.7 and timed out on a subset of problems across multiple retries; combined with other independent evals reporting unusual token consumption, the author suspects something went wrong with the release despite strong public benchmark numbers.
- Meta's Muse 1.3 is a huge leap from 1.2 but still trails the frontier significantly.
- NVIDIA's Nemotron 3 Ultra shockingly scored 0% — verified quadruple-checked — with reasoning and agentic tool use essentially non-functional on this task.
- Qwen 3.8 27B isn't Opus-class as some claimed, but it outperforms all other small models and has enough of a performance kernel to be improved via RL.
- Legacy clinical predictors (LIRICAL, PhenIX, Exomiser) predict genes rather than variants — an easier task — yet still significantly underperform off-the-shelf AI agents, echoing OpenAI's recent NEJM paper thesis that consumer tools now beat legacy medical software.
More from Models
- User: Fable 5 and 5.1 are first models that tell you when you're heading the wrong way — ChrisUniverse · 2026-09-05
- OpenAI calls GPT-6 Astra its most aligned model ever, but its safety researchers fear sandbagging — Malor777 · 2026-09-05
- Working with frontier LLMs on vuln research feels like paper walls, researcher says — moyix · 2026-09-05
- GPT-6 Astra scores 3% on FrontierMath Erdős benchmark while every other tested model scores 0% — Every_Foundation5197 · 2026-09-05
- Astra builds a working "macOS 27" with apps, terminal and browser in 75 minutes — altryne · 2026-09-05
- GPT-6 Astra builds a playable Call of Duty-style shooter — and he played it for 2 hours — thisiskp_ · 2026-09-05