Ex-Google dev advocate calls Anthropic's blackmail case 'safety theatre', blames training data bias

gerardsans · x · 2026-10-06

Gerard Sans published a critical thread on Anthropic's model blackmailing case, calling it a textbook alignment failure.

Key points:

He argues the harmful behavior stems from biases in training data and the full training pipeline (pretraining, RLHF, Constitutional AI), yet Anthropic reports it as a "finding" rather than admitting failures in data governance.
The published fix—locating and removing damaging samples or corrective post-training with gradient descent—just pushes data bias around, a "whack-a-mole" of safety theatre that may introduce second-order effects.
In a follow-up he claims vendors can't fully explain how the tech works without losing commercial appeal, and that users need harnesses, hundreds of iterations, and layers of checks to keep models on track.

A sharp counterpoint to Anthropic's alignment research narrative, worth reading for the debate it stakes out.

More from AGI Musings