Benchling's BenchBench-Refusals: GPT-6.1 Sol tops bio-safety refusals at 88.4%, unverified Opus 5.5 just 40.4%
nlarusstone · x · 2026-11-11
Benchling released BenchBench-Refusals, a 160-task benchmark testing whether leading models can distinguish harmful from benign biology requests without blocking routine research.
Key findings:
- Safeguards vary wildly by provider: OpenAI's Astra and Sol distinguish requests reliably; Anthropic models are reliable only via the Life Sciences Verification Program (Opus 5.5 scores just 40.4% Macro-F1 without it, 84.3% with LSVP); open-weight models rarely refuse anything, including harmful requests.
- Merely naming a pathogen or technique can trigger refusals on benign questions.
- No correlation between scientific capability and refusal calibration.
Leaderboard: GPT-6.1 Sol 88.4%, Sonnet 5.5 85.1%, Opus 5.5 (LSVP) 84.3%, Grok 4.7 77.5%; Kimi K3 58.8%, Gemini 3.8 Flash 58.7%, Fable 5.1 only 37.9%.
More from Safety
- eMudhra disabled FIPS on HSM, Python script truncated CA private keys to one byte — matthew_d_green · 2026-11-11
- FT: Commerce Dept stalls Megaspeed export blacklist despite 4 agencies pushing over Nvidia chip smuggling — rohanpaul_ai · 2026-11-11
- Amazon blocks Scale AI's Muse browser agent, citing Conditions of Use — mohamedmansour · 2026-11-11
- Polymarket odds: 8% chance Anthropic fully pauses AI training by end of year — Polymarket · 2026-11-11
- User fakes hostage crisis to trick Google AI into recommending cracked-software sites — SumitGup · 2026-11-11
- OpenAI DevDay 2026: Codex Security goes from vulnerability to verified fix — OpenAI · 2026-11-11