RespSurvBench: 110 evals on virus surveillance, best model passes just 25.5%

kenbwork · x · 2026-11-06

Researchers introduced RespSurvBench, 110 evaluations spanning diagnostic assays, pathogen discovery, viral evolution, and population surveillance. Across 21 models and 34 model–harness configs, the top pass rate was only 25.5% (Claude Opus 5.5 in Pi), with refusals counted as failures. Anthropic models show notably higher and more variable refusal rates than Gemini/OpenAI/xAI; common failures include treating missing observations as negatives and mistaking artifacts for biological signal. An interactive leaderboard is live on benchmarks.bio.

Related event: RespSurvBench: Top AI Model Passes Only 25.5% on Respiratory Outbreak Monitoring(4 posts)→

Original post →

More from Models

Models channel →