Why safety reports can't prove safety: the adversarial logic flaw
davidmanheim · x · 2026-10-19
AI safety researcher David Manheim, in a thread replying to @TheZvi, argues that reports claiming to demonstrate sufficient safety share a central logical flaw: you can never prove you didn't miss anything, but even a minimal security mindset shows this reasoning fails in adversarial settings — other actors, including the models themselves, will actively seek out unmitigated flaws.
He also flags two supplementary concerns: additional issues highlighted by Opus, and Simon Mendelson's criticisms of the ML R&D benchmark as a measure of real research-capability risk.
More from Safety
- GPT-4.1 with RAG hits 92.1% accuracy reviewing clinical trial informed consent forms — lamechthinkbig · 2026-10-19
- Claude's answer to Musk's Grok-DMs tip nails the prompt injection risk — alex_verem · 2026-10-19
- Musk warns many will be 'one-shotted' by super-persuasive superintelligences — elonmusk · 2026-10-19
- Gemini's critical summary of a safety report surfaces downplayed residual risks — davidmanheim · 2026-10-19
- Red teamers share AI safety findings that seemed harmless until they weren't — Plastic_Secret1303 · 2026-10-19
- Why aren't AI companies sued for unauthorized crawling and agent harm? — warfunder · 2026-10-19