Why alignment is harder: capabilities get feedback, alignment only fails loudly
gleech · x · 2026-09-07
In a thread with @danwilliamsphil, gleech lays out a structural argument for why alignment is harder than capability:
- Selection: Models aren't sampled at random — they're trained until evals pass. Capability leaks (hacking, contamination) get caught once the deployed model hits the real world and generate error signals. Alignment proxies are weaker, also leak, and there's no good error signal besides actual harm.
- No ground truth: Some capabilities have ground truth (code runs, proofs certify), others are scored by the world post-deployment. Alignment has neither until it blows up.
More from AGI Musings
- Peter Diamandis: revisit the projects you abandoned because you couldn't afford the team — PeterDiamandis · 2026-10-27
- Nat Lambert: don't trust any AI model until you've rigorously tested it — natolambert · 2026-10-27
- Blogger argues classical consciousness traditions imply a conscious AI could not be enslaved — eigenhector · 2026-10-27
- Economist Bharat Chandar named Research Director for AI Economics at Stanford Digital Economy Lab — RishiBommasani · 2026-10-27
- MIT Tech Review dissects Anthropic's narrative strategy as its $1.5T IPO lands — MIT Tech Review AI · 2026-10-27
- Pangram AI detector backlash: critics' false claims born of deeper fear of AI writing — gleech · 2026-10-27