AI safety debate: a superhuman model may need just one bit to trigger an avalanche
akbirthko · x · 2026-09-18
In a thread reacting to Noam Brown's AI risk scenario, akbirthko argues that debating any fixed catastrophic scenario misses the point: a clearly superhuman AI could invent novel side-channel attacks, and the real question is whether optimization pressure will push it there. He steelmans Brown's case: what happens inside the receiving system may matter, and perhaps just one bit of information (yes or no) could trigger an avalanche.
More from AGI Musings
- Santa Fe Institute's Eric Smith on the anthropocentrism paradox in one quote — danfaggella · 2026-11-02
- Blogger claims the next generation won't need a college degree to live well — rand_longevity · 2026-11-02
- Carl Frey: 85% in China see AI as beneficial vs 38% in US — resistance maps to expertise costs — carlbfrey · 2026-11-02
- After a painful rebuild, fejau concedes: build on Hermes and you own your AI for life — Teknium · 2026-11-02
- 'Meatwashing': manually rewriting AI output to hide its origin, and why it's a double waste — JeremyNguyenPhD · 2026-11-02
- AI safety researcher: AI companies entirely lack this kind of effective human oversight — davidmanheim · 2026-11-02