RespSurvBench: top AI model scores just 25.5% on respiratory virus surveillance
kenbwork · x · 2026-11-06
RespSurvBench launches with 110 evaluations spanning diagnostic assays, pathogen discovery, viral evolution/transmission, and population surveillance. Across 21 models and 34 model–harness configurations, the best pass rate was 25.5% (Claude Opus 5.5 in Pi), with refusals and execution errors counted as failures.
The accompanying heatmap breaks down refusal rates by evidence source, decision type, and failure class: Claude Fable 5.1 refuses 72% overall, and refusal behavior varies widely within the Anthropic family while Gemini/OpenAI/xAI families are more consistent. Non-exempt models respond to healthcare-record tasks but refuse more on experimental/real-world data.
Common scientific mistakes: treating missing observations as negatives, mismatching records, confusing measurement artifacts with true biological signal. The authors note AI can only become core outbreak-tracking infrastructure if models actually respond. Leaderboard at benchmarks.bio.
More from Models
- Tesla FSD V15 renders real-time 3D street view at night — blurry store signs reveal it's AI-generated — lxfater · 2026-11-23
- Distillation is mostly cope: labs buy astronomical data, dev argues overhyped in local AI community — gnukeith · 2026-11-23
- Sarvam's new 105B checkpoint jumps from 42.6 to 73.9 on benchmarks, claims Flash-Lite Pareto frontier — itsOmSarraf_ · 2026-11-22
- Dev says Google models are making real progress on lossless compression as intelligence proxy — gnukeith · 2026-11-22
- Astra-6.2 'lazy' backlash blamed on user education: default manual mode vs goal mode — alexisgallagher · 2026-11-22
- Musk: Tesla has had a near pixel-perfect world model since last year — giffmana · 2026-11-22