Why the legal system's alignment toolkit fails for AI: copies beat prisons, penalties don't deter
trevposts · x · 2026-11-17
trevposts argues the strongest case that developing superhuman AI is dangerous is that humanity's main alignment tool — the legal system, built over thousands of years of trial and error — doesn't transfer to AI. Incapacitation fails: unlike humans, AIs can be copied and spread across datacenters worldwide, so confinement guarantees nothing. Deterrence fails: negative reward during training isn't something models experience or witness, and penalized behavior can resurface in other forms, so we have no real way to deter an AI from harmful actions.
More from AGI Musings
- Profound CTO dbabbs: software engineering isn't cooked — CS will matter most in the AI era — soleio · 2026-11-17
- Miles Brundage's AVERI publishes key definitions for frontier AI auditing and evaluation — Miles_Brundage · 2026-11-17
- As AI aced homework, calls grow for universities to build AI-free exam infrastructure — _akpiper · 2026-11-17
- AI raised the ambition floor: researcher now lets optimization runs continue past the first win — generativist · 2026-11-17
- Why the AI safety movement is hard to beat: it lures Jane Street traders to $60k Hill jobs — panickssery · 2026-11-17
- CAIS and Scale AI Launch Remote Labor Index to Measure AI Automation of Remote Work — ben_j_todd · 2026-11-17