Underdetermination in Alignment: Training Data Pins Down Capability but Not Goals
gleech · x · 2026-09-07
In a debate thread with @danwilliamsphil, gleech lays out the underdetermination argument: behavior on the training set nearly determines what a model can do (modulo shortcuts and hacking), but barely determines what it wants — many goals are behaviorally indistinguishable. Worse, in the worst case of zero-shot deceptive alignment there's no error signal at all: capability leaks like hacking and contamination get caught in deployment, but alignment-proxy leaks surface only through actual harm.
More from Safety
- Study: AI reviewing AI misses most reward hacking; best models miss 25%+ of hacks — ronbodkin · 2026-10-26
- Fangcun AI, China's first independent frontier AI safety lab, launches with DeepSeek eval report — JacquesThibs · 2026-10-26
- ESR on the Bugpocalypse: locking down frontier models only hands the game to Chinese open weights — colinmcnamara · 2026-10-26
- OpenCode breach exposed AI API keys of 1.4% of Console users — thione · 2026-10-26
- DHS orders DeepSeek products removed from US civilian agency systems — thione · 2026-10-26
- One hacker breached 100+ company websites using AI, scope unknown — thione · 2026-10-26