Nick Bostrom on superintelligence: RL pushes goal-seeking, no blanket AI pause
a16z Podcast · rss · 2026-10-11
Philosopher Nick Bostrom joins the a16z podcast to revisit superintelligence and existential risk now that capable AI systems exist. Key points:
- Reinforcement learning may push models toward stronger goal-seeking behavior; recent agent experiments inform views on instrumental convergence.
- He favors preserving the ability to slow frontier development rather than committing now to a broad or indefinite AI pause.
- AI risks should be weighed against upside: medicine, longer lifespans, new ways of being human.
- Also covers machine consciousness, open-source tradeoffs, original philosophy by machines, mind uploading, and the simulation hypothesis.
More from AGI Musings
- Amazon paper: LLM as research world model lifts experiment ranking correlation from 0.506 to 0.774 — rohanpaul_ai · 2026-10-11
- Terence Tao's Math 2.0 lecture uses thalidomide case to argue results can outrun comprehension — burny_tech · 2026-10-11
- AI alignment thought experiment goes meme: hidden supervirus in the cancer cure — basedjensen · 2026-10-11
- Self-taught 17-year-old with no degree lands 5 ~$1M+ offers from top SF AI labs — deedydas · 2026-10-11
- Cognitive scientist Andrew Lampinen defends inferring mental states from behavior in LLM debate — AndrewLampinen · 2026-10-11
- Redditors debate: when will a normal PC run Sonnet-level AI locally? — JumpAppropriate714 · 2026-10-11