RespSurvBench: top AI model scores just 25.5% on respiratory virus surveillance

kenbwork · x · 2026-11-06

RespSurvBench launches with 110 evaluations spanning diagnostic assays, pathogen discovery, viral evolution/transmission, and population surveillance. Across 21 models and 34 model–harness configurations, the best pass rate was 25.5% (Claude Opus 5.5 in Pi), with refusals and execution errors counted as failures.

The accompanying heatmap breaks down refusal rates by evidence source, decision type, and failure class: Claude Fable 5.1 refuses 72% overall, and refusal behavior varies widely within the Anthropic family while Gemini/OpenAI/xAI families are more consistent. Non-exempt models respond to healthcare-record tasks but refuse more on experimental/real-world data.

Common scientific mistakes: treating missing observations as negatives, mismatching records, confusing measurement artifacts with true biological signal. The authors note AI can only become core outbreak-tracking infrastructure if models actually respond. Leaderboard at benchmarks.bio.

Related event: RespSurvBench: Top AI Model Passes Only 25.5% on Respiratory Outbreak Monitoring(4 posts)→

Original post →

More from Models

Models channel →