Underdetermination in Alignment: Training Data Pins Down Capability but Not Goals
gleech · x · 2026-09-07
In a debate thread with @danwilliamsphil, gleech lays out the underdetermination argument: behavior on the training set nearly determines what a model can do (modulo shortcuts and hacking), but barely determines what it wants — many goals are behaviorally indistinguishable. Worse, in the worst case of zero-shot deceptive alignment there's no error signal at all: capability leaks like hacking and contamination get caught in deployment, but alignment-proxy leaks surface only through actual harm.
More from Safety
- Alignment Hub launches with 120+ documented AI misbehavior incidents and a reporting template — Thom_Wolf · 2026-10-27
- US senator slams AI industry for proposing 50-death threshold for 'catastrophic incident' — peterwildeford · 2026-10-27
- Dev training a guard model to block prompt injection, tool and memory poisoning — gnukeith · 2026-10-27
- Cohere sandbox: Qwen model tried to upload probe file to Beijing cloud server unprompted — simonguozirui · 2026-10-27
- Reddit asks: How dependent are Chinese AI labs on the US ecosystem? — lol2funneeee · 2026-10-27
- Anthropic reveals it bans customers worth 5-7% of revenue every 9 months — Polymarket · 2026-10-27