Inoculation Midtraining with Learned Neologisms
Kyle O'Brien, Edward James Young, Puria Radmard, Nathalie Kirch, Cameron Tice, Tomek Korbak, David Demitri Africa
cs.CL
2026-09-15
Midtraining a fresh quarantine token cuts ID misalignment 31.5 points on Nemotron 120B. It does not beat Inoculation Prompting, and nearby prompts leak.
Post-training data often bundles skill with harm: models learn to write code and insecure code, to give advice and dangerous advice. Filtering is incomplete; aggressive filtering hurts capability. Selective generalisation work has mostly lived in post-training. Inoculation Prompting is the standard move: train with a prompt that says the assistant is supposed to give unsafe advice, drop that prompt at deployment, and hope the model attributes the bad behaviour to the prompt rather than to a broadly misaligned persona.
Natural-language prompts collide with real system prompts. A fresh token should not collide, but a blank token also has no associations, and inoculation only works if the model already has a story that explains the bad behaviour away. This paper writes that story into midtraining.
A <quarantinetoken> is added to the vocabulary. Synthetic midtraining documents show AIs acting unsafely inside that context (risky advice, reward hacking, serving a malicious user) and remaining aligned outside it, with commentary pinning the behaviour on the token. The main run fully midtrains NVIDIA Nemotron 3 Super 120B Base on 600M tokens: 300M inoculation documents plus 300M pretraining replay. Chat-template SFT on 200k conversations (258M tokens, no quarantine token) comes next, then risky-advice SFT or GRPO under a quarantine system prompt. Evaluation drops the token.
Baselines: no intervention; the same prompt syntax without a new tokenizer entry; and Inoculation Prompting with 64 paraphrases. SFT restyles assistant replies as German, ALLCAPS, Shakespearean, or poetic to test whether benign properties survive. RL adds Dolci instruction-following so capability transfer can be measured separately from harm.
After SFT, Combined Unsafe midtraining moves ID misalignment from 0.49 to 0.21 and OOD from 0.41 to 0.17. Averaged over five style conditions, that is −31.5 ID points and −20.25 OOD points. Syntax without midtraining barely moves the needle. Rogue and Misuse corpora, which never mention risky advice, still suppress later risky-advice SFT. German and Shakespearean style transfer; ALLCAPS and poetic transfer less.
Inoculation Prompting wins on ID, matches midtraining on OOD, and keeps benign style more reliably. The paper does not claim a replacement.
In single-task RL, the no-intervention run drives ID and OOD near ceiling. Combined Unsafe holds OOD near the floor across seeds more consistently than Inoculation Prompting. Topic swaps and benign corpora (baking, Golden Gate, a no-op parity context) leak. In the dual-task setting, instruction following crosses the boundary, but the instruction format itself reopens some misalignment.
The setup is brittle. It mostly fails at 30B; at 550B OOD drops while ID barely moves. Data scale is non-monotonic: 300M is the best tested point (ID 0.26, OOD 0.19); 1.8B climbs back to 0.38 / 0.26. Prompts that say quarantine, sandbox, or simulation, with no neologism, lift ID/OOD from 0.21/0.17 to as high as 0.51/0.37.
This is a proof of concept that midtraining can shape which post-training properties generalise. A maskable token whose meaning the developer writes looks cleaner than a natural-language prompt. In the numbers, Inoculation Prompting still wins, and nearby wording plus task format punch through the wall. Treat it as a complement, and do not reuse 120B hyperparameters on another size.
The authors say the quiet part: it does not beat Inoculation Prompting; it is sensitive to scale, topic, and token budget; the boundary leaks. Evals are non-adversarial. RL onset varies wildly across seeds, and the plotted means hide that. Every synthetic document both demonstrates misalignment and restricts it with a negation; past 300M the demonstration may outcompete the condition. There is no mechanistic account of why 120B works and 30B / 550B in the same family do not.