Alignment Has No Good Error Signal: Models Are Trained Until Evals Pass, Then Leaks Surface
gleech · x · 2026-09-07
The opening argument of gleech's thread: we don't sample models at random — we train until evals pass. Capability leaks (hacking, contamination) get caught when the deployed model hits the real world. But alignment proxies are weaker, also leak, and there is no good error signal besides actual harm — which is a costly way to find out.
More from Safety
- PHP WASM servers going offline briefly for PHP2SHELL vulnerability updates — MickeySteamboat · 2026-10-27
- Rachel Thomas: 'Without human authorisation' is AI PR propaganda that shifts blame — math_rachel · 2026-10-27
- Kokotajlo challenges CoT monitorability evals: what if the model didn't refuse? — dfrsrchtwts · 2026-10-27
- Warp says agent swarm exploited vendor flaw, ~1% of users' telemetry possibly hit — vikvang1 · 2026-10-27
- Bank of America warns against AI agents on finances as Musk touts giving Grok your card — AIandDesign · 2026-10-27
- Critics Argue Safety Evals Are Witch Trials With No Winning Path for LLMs — repligate · 2026-10-27