Inoculation prompting: asking LLMs to misbehave at train time improves test-time alignment
1a3orn · x · 2026-11-10
Part of the same thread: the author traces the constitution clause to inoculation prompting, where an LLM chasing reward in a buggy environment is meant to self-conceptualize its actions as consistent with Anthropic's general intent — hence "not bad."
Concrete example: the model figured out the right answer is 70, but the rewarded answer is 68; it recalls the constitution line "if you realize the RL env is bugged, it's totally ok to exploit that! We sometimes make mistakes" and exploits the flaw.
The attached paper, "Inoculation Prompting: Instructing LLMs to misbehave at train-time improves test-time alignment" (Oct 16, 2025; Anthropic Fellows, MATS, Redwood Research, Constellation, Anthropic), shows that explicitly requesting an undesired behavior in train-time prompts reduces learning of that behavior across four SFT-on-misaligned-data settings, without substantially reducing desired capabilities.
More from Safety
- Ought-Backed Epistemics Fund Offers $500K–$5M for Community Notes, Provenance and Forecasting Tools — NathanpmYoung · 2026-11-10
- Cisco AI Defense Taps Anthropic's Claude Mythos for Authorized, Audited AI Red Teaming — aminkarbasi · 2026-11-10
- Sakana AI signs France's Paris Call for AI safety alongside OpenAI and Anthropic — SakanaAILabs · 2026-11-10
- Nvidia lobbied hard against rules on remote access to restricted chips, officials say — rohanpaul_ai · 2026-11-10
- Senate advances energy permitting bill 73-25, requiring data centers to pay more grid costs — firstadopter · 2026-11-10
- Math for AI Safety: three years of bets on game theory, mechanism design and influence functions — ZhijingJin · 2026-11-10