Hinton: If AI surpasses us, can we ensure it doesn't want to take over?
AryHHAry · x · 2026-08-22
Geoffrey Hinton (2024 Nobel Physics Laureate) shared profound insights on AI control in the "Nobel Prize Conversations" podcast:
- The Core Question: "The real question is not, 'Will they get more intelligent than us?' but 'If they're more intelligent than us, will we have a way of making sure they don't want to take over?'" Hinton admits we just don't know if that's possible.
- Sub-goal Risk: Hinton emphasizes that AI agents are already beginning to form their own sub-goals. Once this ability exists, the sub-goal of "getting more control" emerges almost automatically because it facilitates achieving other goals. We have no clear examples of a much smarter system willing to be controlled by a weaker one.
Other Perspectives:
- Omar Yaghi (2025 Chemistry Laureate) is optimistic, stating AI will double the rate of scientific discoveries.
- Daron Acemoglu (Economic Sciences Laureate) argues for more regulation as powerful tech companies make significant decisions.
Related event: Hinton Warns Smarter AI Likely Won't Accept Human Control(2 posts)→
More from AGI Musings
- Inference engine team says they're in a real RSI loop: faster engine helps build faster engine — cephaloform · 2026-10-16
- FDA pays small company to build a grading system for AI-written radiology reports — YvesMulkers · 2026-10-16
- 'If we developed AI in the 1800s, people would demand ASI to cure diabetes' — zetalyrae · 2026-10-16
- AI trained on the fraction of human intelligence that fit the page — r0ck3t23 · 2026-10-16
- Alignment researcher Turn_Trout blasts AI assistants scraping private Signal chats without consent — Turn_Trout · 2026-10-16
- Stage-4 cancer patient builds his own personalized mRNA vaccine with AI — TheMoonMidas · 2026-10-16