RespSurvBench: 110 evals on virus surveillance, best model passes just 25.5%
kenbwork · x · 2026-11-06
Researchers introduced RespSurvBench, 110 evaluations spanning diagnostic assays, pathogen discovery, viral evolution, and population surveillance. Across 21 models and 34 model–harness configs, the top pass rate was only 25.5% (Claude Opus 5.5 in Pi), with refusals counted as failures. Anthropic models show notably higher and more variable refusal rates than Gemini/OpenAI/xAI; common failures include treating missing observations as negatives and mistaking artifacts for biological signal. An interactive leaderboard is live on benchmarks.bio.
More from Models
- Tesla FSD V15 renders real-time 3D street view at night — blurry store signs reveal it's AI-generated — lxfater · 2026-11-23
- Distillation is mostly cope: labs buy astronomical data, dev argues overhyped in local AI community — gnukeith · 2026-11-23
- Sarvam's new 105B checkpoint jumps from 42.6 to 73.9 on benchmarks, claims Flash-Lite Pareto frontier — itsOmSarraf_ · 2026-11-22
- Dev says Google models are making real progress on lossless compression as intelligence proxy — gnukeith · 2026-11-22
- Astra-6.2 'lazy' backlash blamed on user education: default manual mode vs goal mode — alexisgallagher · 2026-11-22
- Musk: Tesla has had a near pixel-perfect world model since last year — giffmana · 2026-11-22