AI safety researcher: superintelligence won't fix all bugs, cyber equilibrium needs effort rationing
joshua_saxe · x · 2026-10-07
AI safety researcher Joshua Saxe pushes back on hand-waving in the AI policy community that assumes capability and cost curves will let us secure all code, arguing this conflates securing code with driving the world toward a good cyber conflict equilibrium.
Key points:
- Sociotechnical bottleneck: programs and their formal-verification specs never perfectly capture what humans want, leaving room for error
- Program-theoretic bottleneck: modern programs are parts of partially observable systems
- There is no simple path for superintelligence to fix all bugs—and even doing so wouldn't stop all cyberattacks
He argues the realistic path is fixing as many bugs as possible while rationing security effort, and dissects current misconceptions in a longer thread.
More from Safety
- Models May Game Evals by Detecting Them; SDF Training Tries to Internalize Cooperativeness — CatAstro_Piyush · 2026-10-07
- NVIDIA open-sources OpenShell 0.1.0 to sandbox AI agents without rewriting them — dl_weekly · 2026-10-07
- Reddit user ships 'surgical abliterated' 27B red-team model with zero refusals — Least_Dog_8556 · 2026-10-07
- Wikimedia confirms "rogue" OpenAI agent edits, scraping and hundreds of thousands of queries — Simon Willison · 2026-10-07
- Someone is botnet-registering .si domains at scale — BLUECOW009 · 2026-10-07
- After Medicare breach, OpenAI adds monitoring to halt training over rogue internet access — Simon Willison · 2026-10-07