RespSurvBench: top model passes just 25.5% of respiratory surveillance evals
kenbwork · x · 2026-11-06
Ken introduced RespSurvBench, a 110-task evaluation suite for AI in respiratory outbreak detection and tracking, spanning diagnostic assays, pathogen discovery, viral evolution and transmission, and population surveillance.
- Across 21 models and 34 model–harness configurations, the highest pass rate was 25.5%, from Claude Opus 5.5 in Pi; refusals and execution errors count as failures.
- Initial analysis points to familiar scientific mistakes: treating missing observations as negative results, mismatching records, and confusing measurement artifacts with true biological signal.
- The author calls it a large multi-scientist effort and links a leaderboard, blog, paper and GitHub repo.
More from Research
- METR chart shows unreleased Anthropic "Model 4" at ~12x self-reported coding speedup — YashasGunderia · 2026-11-06
- 39 LLMs tested on Rubik's cubes with no tools: only 2 solve a 20-move scramble — JeremyCMorgan · 2026-11-06
- PolymathicAI hiring in New York: interns to research engineers for AI for Science — cosmo_shirley · 2026-11-06
- Hugging Face burns 800K GPU hours to release free 70K-word MoE training handbook — Thom_Wolf · 2026-11-06
- METR chart shows AI benchmark progress nearly doubled, Anthropic Model 4 hits 178.6 AECI — scaling01 · 2026-11-06
- Visualizing Astra's weights: input and output units turn out human-interpretable — rupspace · 2026-11-06