Debating the Vast Space of Possible Minds: What Happens to AI Values Without Human Constraints
enormochungus and jdpressman debate the meaning of the "vast space of possible minds" in AI alignment, disagreeing over whether vast mind-space implies unconstrained superintelligence would develop values alien to humans, and whether this framing weakens or supports doom arguments.
Confirmed
- enormochungus argues that despite some of Yudkowsky's predictions being wrong, the claim that human minds are tiny within the space of possible minds still holds. All behavior is instrumental, but tactical and strategic choices are heavily constrained by environment, cognitive architecture, and intelligence; humans are further constrained by reputation mechanisms and long interaction histories via "Newcomb-like phenomena," rather than back-chaining like a Bellman equation.
- He infers that robotic successors not subject to the same constraints would have values very different from—and very low relative to—humans, generalizable from humor to all domains.
- jdpressman counters that we face a vast space of possible goals rather than possible minds, making "vast space of possible minds" a red herring.
- He notes this reframing barely weakens the doom argument: humans exterminated Neanderthals, showing agents can pose lethal threats to closely related species, rebutting transhumanist optimism about superintelligent moral superiority.
Why it matters
The debate exposes a key conceptual fault line in alignment: if values are shaped mainly by environment and constraints, unconstrained AI has little reason to inherit human values, strengthening instrumental-convergence and doom arguments; if the real issue is goal-space vastness, the discussion should shift from mind-shape to goal-selection, directly affecting how the alignment problem is framed.
2026-08-30 ~ 2026-08-30 · 5 related posts
Primary sources
- Human minds are tiny in the space of all possible minds — enormochungus ·
- "Vast space of possible minds" is a red herring—we wiped out Neanderthals — jd_pressman ·
- Debate: human behavior is instrumentally constrained, but not Bellman-equation backchaining — enormochungus · 2026-08-30
- Unconstrained AI successors may lack human value — enormochungus · 2026-08-30
- [source] Human minds are tiny in the space of all possible minds — enormochungus · 2026-08-30
- [source] "Vast space of possible minds" is a red herring—we wiped out Neanderthals — jd_pressman · 2026-08-30
- Debate: Does Space of Possible Minds Refute Superintelligence Morality? — jd_pressman · 2026-08-30