New respiratory-virus agent benchmark tops out at 30.2% pass rate
kenbwork · x · 2026-11-14
A new agentic benchmark, BioSecBench-RespiratoryVirus-Countermeasures, extends prior respiratory-surveillance work from threat detection to understanding and treating disease.
- 64 evaluations across five groups: viral structure/function, immunity and antigenic variation, host response and disease biology, antiviral activity and resistance, and clinical/population outcomes.
- Three public tasks are available for inspection, covering antibody occupancy, Omicron neutralization, and vaccine safety reporting.
- Leaderboard: Gemini 4 Carbon with Pi leads at 30.2% overall pass rate, followed by Claude Sonnet 5.5 with Pi (18.8%) and Claude Fable 5.2 with Claude Code (17.7%). Refusals and errors count as failures.
- Scores are low across the board, and the author notes the causes are not solely model capability — highlighting how hard this domain remains for frontier agents.
More from Models
- Anthropic rumored to drop massive multi-field science breakthroughs around its IPO — Dr_Singularity · 2026-11-14
- GraphSAGE author tkipf endorses Gemini-orchestrated Carbon+Hydrogen as best agent setup — tkipf · 2026-11-14
- Fable 5.2 hits 41.25% on Remote Labor Index, doubling Opus 5.5 in 48 days — ChrisGPT · 2026-11-14
- Claude Duet Invites All US Waitlisted Pro/Max/Ultra Users, With Limit Resets — lydiahallie · 2026-11-14
- Sleeper agents in open-weight models are overblown, argues researcher: closed APIs are riskier — adam_karvonen · 2026-11-14
- Fable 5.2's Edge Over Opus 5.5 Grows With Task Length: 93% vs 85% at 16+ Turns — arena · 2026-11-14