Anthropic Study: Fine-Tuned Lie Detectors Fail to Generalize OOD
PandaAshwinee · x · 2026-08-24
Anthropic published an alignment science study investigating whether training a dedicated deception monitor offers benefits beyond simply prompting a capable model.
Key Findings:
- In-distribution: Fine-tuned lie detectors outperform prompted baselines.
- Out-of-distribution (OOD): The advantage mostly vanishes. Fine-tuned detectors barely beat prompted baselines, and larger prompted models often outperform them entirely.
- Larger models are generally better at detecting lies, though the trend is not monotonic.
Context:
- Reliable lie detection raises the cost of hiding misalignment but does not align the model itself.
- Prior work (e.g., Liars' Bench) showed detectors fail to generalize across lie types.
Methodology:
- Elicited on-policy lies from open-weight models across 12 settings.
- Fine-tuned models on a binary classification task (did it lie?) and evaluated generalization by training on half the lie types and testing on the others.
More from Safety
- Proof launches x401, first protocol letting AI agents prove who they represent — csuwildcat · 2026-10-15
- SK Hynix ties new US plant to looser chip equipment exports to China, sources say — pstAsiatech · 2026-10-15
- Fox-IT releases the Cobalt Strike Beacon Corpus for detection research — cyb3rops · 2026-10-15
- Bank of America puts 4,000 people on fixing vulnerabilities uncovered by AI models — The Decoder · 2026-10-15
- Inside SAIF: a16z's $25M bet on AI rewriting the economics of cyber defense for small towns — a16z · 2026-10-15
- a16z launches SAIF, a $25M nonprofit to bring AI cyber defense to 1,800 rural hospitals — a16z · 2026-10-15