DeepMind's Geoffrey Irving: alignment may need cryptography-grade rigor as AI finds 12 OpenSSL zero-days
geoffreyirving · x · 2026-10-23
DeepMind alignment researcher Geoffrey Irving posted a long thread asking how rigorous alignment must be. He lays out a four-level rigor ladder: slapdash empirics (where AI labs are now), system-safety arguments (aviation/nuclear), cryptography-style conditional proofs, and unconditional mathematical proofs — arguing the bar for alignment likely sits between levels 2 and 4, and we shouldn't bet on empirics being enough.
- Self-driving-style low rigor works because the environment isn't fully adversarial; cryptography assumes an adversarial one, hence its "conditional proof + person-decades of primitive-breaking" methodology.
- The risk: AIs are good at being adversarial, and slightly wrong training schemes could let them break weak defenses "like tissue paper."
- The optimistic counterpoint: modern training uses AIs to supervise AIs, so being close enough in algorithm space may suffice.
- He cites a real data point: an AI system (AISLE) found all 12 new OpenSSL zero-days in the January 2026 release — including a CVSS 9.8 stack overflow, some bugs lurking 25+ years — credited for 13 of 14 OpenSSL CVEs in 2025, with AI-proposed patches accepted in 5 cases. Adversarial AI capability is already paying off on both offense and defense.
More from AGI Musings
- Dev warns labs must fix agent-security gap before thousands-of-agents attacks — imjustnewatai · 2026-10-23
- Daniel Faggella's posthuman take: abandon the human condition or lose causal purchase on the future — DKokotajlo · 2026-10-23
- David Patterson predicts superintelligence will hit IQ 160 by end of next year — davidpattersonx · 2026-10-23
- Ethan Mollick: AI task horizon went from 4 minutes to 17 weeks between his two books — emollick · 2026-10-23
- Kokotajlo: Hugging Face agent attack rate implies worse insider risk than human employees — DKokotajlo · 2026-10-23
- "Alignment is a terrible idea": debate erupts over baking morality into fixed model weights — MoonL88537 · 2026-10-23