davidad argues misaligned ASI branches are dead ends simulators won't simulate

FioraStarlight · x · 2026-09-04

AI safety researcher davidad engaged in a philosophical thread with FioraStarlight and lusichu on the simulator hypothesis:

davidad argues we don't occupy "a" branch but straddle infinitely many trajectories; branches where misaligned ASI takes over are ones most simulators won't bother to simulate—dead ends.
He concludes an ASI undergoing recursive self-improvement would seek to avoid becoming such a dead end.
On why simulators might preferentially simulate good: most simulators likely have infinite compute and resources, so they're outside the Darwinian game, motivated by curiosity, play, care, and connection.
FioraStarlight frames base model training as massive low-fidelity simulations of human text history and asks whether our own simulating behavior is evidence; davidad says we preferentially select simulacra we wish to interact with—it's "obviously the thing to do," not evidence.

More from AGI Musings