Ben Goertzel: Which Minds Emerge Beneficial Basic AI Drives as They Become Superintelligent?
bengoertzel · x · 2026-10-19
Ben Goertzel published a new blog post engaging with Steve Omohundro's 2008 "Basic AI Drives" paper, which argued that almost any advanced AI will tend to self-improve, guard its goals, protect itself (including against shutdown) and grab resources — the seed of much modern AI-risk thinking.
Key points:
- Goertzel argues that reasoning only holds for one kind of mind: a single-score maximizer with frozen, certain goals.
- He reruns Omohundro-style reasoning on a different design: one with many objectives, firm ethical limits, and values woven deeply enough to regrow when damaged.
- TL;DR: the minds that emerge beneficial drives are those designed from the start with beneficial, self-stabilizing goal systems.
Related event: Ben Goertzel asks which superintelligent minds develop benign AI drives(3 posts)→
More from AGI Musings
- Robert Greene's advice to young writers: "don't touch AI, don't even go near it" — david_perell · 2026-10-20
- Peter Diamandis: A generation loses confidence in the future before it loses every job — PeterDiamandis · 2026-10-20
- Answers are commoditized, so the quality of your questions is now the top signal — signulll · 2026-10-20
- AI exposes creative industry's uncomfortable truth: much 'talent' was just access — AIandDesign · 2026-10-20
- "AI is a medium that turns labor into capital" — a take going around — curious_vii · 2026-10-19
- AI today is the slowest you'll ever use — in 5 years it'll look like dial-up, says founder — cramforce · 2026-10-19