Superintelligence and provable security: can any computer resist a smarter AI?
jessi_cata · x · 2026-11-18
A philosophical AI-safety debate between jessicata and enormochungus on whether higher intelligence always retains an advantage in conflicts.
- jessicata argues it seems impossible to build a supercomputer no superintelligence could destroy or hack: the relevant security property is likely not provable, and even without provable security, gains from better hacks keep rewarding more intelligence.
- enormochungus counters abstractly: computation needs a substrate, so security is bounded by min(secure computation, secure substrate) — the former saturates, the latter is a physics question, and the substrate isn't inexhaustible.
They also discuss why nukes are a poor analogy (they work because agents cluster on Earth; spread over light-years they're easier to counter), true vacuum collapse and black-hole collapse scenarios, whether information could cross a vacuum-collapse boundary panspermia-style, and enormochungus concedes the argument is wrong on reflection, though perhaps not for the reason expected.
More from AGI Musings
- zetalyrae: ASI debates are bait-and-switch, spanning 'next year' to 'Dyson sphere' — zetalyrae · 2026-11-18
- Can superintelligence get past a wall of dumb rocks? An AGI speculation thread on Turing completeness and security proofs — jessi_cata · 2026-11-18
- 'Permanently unemployable' rhetoric sparks pushback: we still have agency over AI's future — kylekabasares · 2026-11-18
- Does intelligence level n build a 'super lance' that 10n can't beat? — jessi_cata · 2026-11-18
- Judea Pearl: econometricians don't even understand potential outcomes themselves — yudapearl · 2026-11-18
- Modeling AI takeoff with Fable 5.2: lab metrics run 4.7 years ahead of real value — ronbodkin · 2026-11-18