Safety Mid-Training paper cuts attack success rate 72-81% without hurting capability
stanfordnlp · x · 2026-10-26
A new arXiv paper by Zhengyue Zhao, Liwei Jiang, Yejin Choi and Chaowei Xiao introduces "safety mid-training," a stage between pre-training and post-training that jointly develops safety and general reasoning through a shared reasoning structure.
Key ideas
- Safety alignment is usually done in post-training; safety can degrade when later training drops safety data, and safety gains often cost general capability.
- The authors treat safety as a capability learnable via the structure shared with general tasks: understanding intent, applying knowledge, reasoning about consequences.
- Method: have the base model generate structured documents and reasoning traces from safety and utility prompts, then use them for mid-training.
Results
- Average attack success rate drops 72%–81% relative to Qwen3-1.7B/4B/8B base models, while general capability improves.
- Benefits persist through post-training: after general-only SFT, mid-trained models retain lower ASR than baselines without sacrificing capability, and stay safe under general-only GRPO with no safety rewards.
More from Safety
- Georgia officials: Anthropic-linked bot scraped public voter data, didn't touch registration system — asusarla · 2026-10-26
- PHP WASM servers show no sign of PHP2SHELL impact so far, patch in progress — MickeySteamboat · 2026-10-26
- Apple's week-long curl/git CA bundle lag sparks developer security debate — kingslyj · 2026-10-26
- LLMs and the Law: training doesn't make a model a derivative work, FAQ explains — esrtweet · 2026-10-26
- OpenAI's Daybreak Blue isn't hard for individuals to get — just set up a hardware key 2FA — pvncher · 2026-10-26
- Research agenda for multi-agent swarms: 7 open alignment problems — ZhijingJin · 2026-10-26