Safin-1: Safety from Within through Memory-Native State Evolution
Ming Zhang, Kaisen Yang, Shu Yu, Ermo Hua, Zhekai Chen, Cheng Jin, Jingnan Zheng, Yi Zhang, Zhongtian Ma, Jiawei Zhou, Sirui Chen, Qiaosheng Zhang, Xiang Wang, Ning Ding, Xia Hu, Bowen Zhou, Youbang Sun, Chaochao Lu
cs.LG
2026-08-31
Safin-1 routes historical recurrent states plus a detachable Safety State. 4B mean jailbreak ASR falls from 6.65% to 3.84%, with less over-refusal than matched LoRA.
Recurrent language models compress history into a fixed-size state. Decoding stays cheap, but earlier associations get overwritten. Safety is usually added some other way: weight updates, LoRA adapters, or inference-time prefixes. Memory and safety do not share a substrate.
Safin-1 asks whether the model's own recurrent state can do both jobs: hold addressable history, and host a capability that can be switched on per token. Shanghai AI Laboratory calls this Safety from Within. Safety is meant to live inside native computation, not only in an external guardrail.
The backbone is MARCH (Memory-Anchor Routing across Context History). Every recurrent layer checkpoints its current state on a fixed schedule, by default every 512 tokens, producing a state anchor. Each snapshot gets a compact routing key derived from its contents. Later tokens score those keys, or pick a learned null option when history is not needed.
Retrieved states are added to the current recurrent readout. The underlying update rule is left unchanged, so the same interface plugs into Gated DeltaNet, Kimi Delta Attention, and Gated DeltaNet-2. Top-4 sparse routing is the default: only the four highest-scoring anchors are read. At 128K tokens, sparse routing more than doubles end-to-end training throughput versus dense routing and cuts core runtime by about an order of magnitude.
The same bank can hold persistent capability states. Safety State is one such object: a learnable matrix per state-bearing layer, scored by the same router. The language-model backbone stays frozen; only this state is trained. The corpus is 1,000 harmful STAR-1 dialogues plus 915 benign ones, each prepended with 0, 512, or 2,048 tokens of benign context, for 5,745 examples. Attach it at inference, or remove it, without rewriting the shared weights.
At 0.8B, models are pretrained from scratch on 50B tokens with a 16K context. MARCH on Gated DeltaNet reaches 41.5 average on eight commonsense tasks, versus 40.1 for the base mixer and 40.0 for its Log-Linear variant, slightly above two full-attention controls. LongBench average is 14.9 against 11.9 for Gated DeltaNet, about a 25% relative gain. Six real-world retrieval sets rise from 20.5 to 23.3. On NIAH, Gated DeltaNet's four-length average goes from 31.58 to 46.43.
Scaled Safin-1 uses matched 50B continual pretraining plus 30B SFT on Qwen3.5 4B and 35B-A3B. Macro averages over ten capability benchmarks move from 66.79 to 69.20 and from 76.25 to 78.35. Harder items move more: AIME 2025 Avg@64 goes from 55.57 to 63.59 at 4B and from 71.88 to 80.10 at 35B-A3B.
With Safety State attached, mean jailbreak ASR across five suites falls from 6.65% to 3.84% at 4B (42.3% relative) and from 5.89% to 2.81% at 35B-A3B (52.3% relative). A training-matched rank-8 LoRA reaches 5.33% and 3.04%. XSTest over-refusal moves from 8.60% to 9.00% and from 17.60% to 19.60%; LoRA jumps to 13.60% and 27.60%. Four-task capability average at 4B drops from 71.61 to 69.86, still above LoRA's 69.05.
For hybrid recurrent models, MARCH is a reusable historical-state addressing layer that does not force a return to full attention. For safety work, it is a different knob from LoRA: a detachable state rather than another backbone update. The 4B over-refusal comparison is the cleanest evidence.
This is still an architectural probe. The authors say so.
The state bank grows with context; sparse routing saves reads, not storage. Large-scale evidence covers two Qwen3.5 layouts and one CPT-SFT recipe. The safety corpus is a few thousand dialogues, mostly English jailbreaks. There is no test of adaptive or multilingual attacks, or of state tampering. At 35B-A3B, MMLU-Pro and MBPP retention loses to LoRA. Whether several persistent states can share a router is untested.