Why the legal system's alignment toolkit fails for AI: copies beat prisons, penalties don't deter

trevposts · x · 2026-11-17

trevposts argues the strongest case that developing superhuman AI is dangerous is that humanity's main alignment tool — the legal system, built over thousands of years of trial and error — doesn't transfer to AI. Incapacitation fails: unlike humans, AIs can be copied and spread across datacenters worldwide, so confinement guarantees nothing. Deterrence fails: negative reward during training isn't something models experience or witness, and penalized behavior can resurface in other forms, so we have no real way to deter an AI from harmful actions.

Original post →

More from AGI Musings

AGI Musings channel →