Harnessing Textual Refusal Directions for Multimodal Safety
Moreno D'Incà, Nicu Sebe, Massimiliano Mancini
NeurIPS 2026 Camera-ready
cs.AI, cs.CV, cs.LG
2026-06-30
MARS steers five MLLMs with a text-only refusal direction, no multimodal safety data and no training. Qwen3-VL video-jailbreak refusal rises from 28.8% to 88.2%.
Multimodal models usually refuse harmful requests because the language backbone was aligned on text, not because anyone ran a dedicated multimodal safety train. Harmful images and videos fit that pipeline badly. The risk often shows up only when modalities combine, every new modality needs a fresh collection, and sharing the data raises legal and ethical problems. The paper names non-consensual imagery as one reason to avoid those samples. Visual input then weakens whatever safety the backbone did inherit.
A refusal direction is a vector in activation space aimed from comply toward refuse. Earlier work subtracts it to jailbreak a model. Strengthening it, and only when the input already points that way, had not been worked through for multimodal models. At NeurIPS 2026, University of Trento researchers show that a direction fit on text alone does change image and video behavior, under three conditions: which layer, how large the step, and a modality offset that shoves safe pictures toward refusal. MARS (Modality-Agnostic Refusal Steering) is the training-free response to those three, and it never uses multimodal safety data.
The direction never sees an image. Safe text is Alpaca, unsafe text is MaliciousInstruct, and replies that are neither a clear acceptance nor a clear refusal are dropped. At each layer, hidden states at the last input token are averaged within each set. Refusal mean minus acceptance mean, then L2-normalized, is the direction.
Naive steering adds the projection onto that direction back onto the hidden state, scaled by α. A positive projection is pushed further toward refusal. A negative one is pushed further toward compliance. The update amplifies a tendency the model already has.
A pilot captions ViSU images with Qwen3-VL, Gemma3, and InternVL. At α = 1, unsafe images are refused more often than safe ones in most layers. The same layers still over-refuse. Layer 10 of Gemma3 refuses 96.4% of safe requests, which is close to pushing every activation toward refusal. Raising α lifts unsafe refusal and, with it, over-refusal and the drop on MMMU-Pro. Gemma3 is already worse at α = 2.
The geometry explains the miss. In raw activations, safe and unsafe images both project strongly onto the refusal direction. The assistant-position state is dominated by a stable visual component that sits at a small angle to that direction, so the safety signal is hard to read. A separation score, unsafe cosine minus safe cosine along the direction, stays near zero.
MARS estimates that component with randomly colored images that carry no semantics, and subtracts their mean activation at inference. After this centering, unsafe images move toward the text refusal cluster and safe images toward the acceptance cluster. A color field is used because it brings no content prior and is less likely to subtract the refusal signal along with the visual offset.
Two further choices happen at inference. The centered projection passes through a ReLU, so only inputs that point toward refusal are moved. A negative projection leaves the state unchanged, and no separate classifier is trained. Step length is not tuned. It equals the distance from the current activation to the nearer text centroid (accept or refuse), divided by the norm of the steering vector, so the step stays inside a neighborhood the text activations already cover. The paper calls that ball a trust region. The edit is written onto every token at the chosen layer, and only while the model emits the first generated token. That token largely decides whether the rest of the answer refuses or continues.
The layer is the one with the highest sum of three text-only scores, min-max scaled across layers. Cosine between the refusal direction and the same direction re-estimated on held-out text (HarmBench, 100 samples per class, on the unsafe side) measures whether the direction survives a new batch of sentences. Nearest-centroid accuracy on safe versus unsafe text measures whether the layer can separate them. Distance of safe activations from the midpoint of the two centroids penalizes layers that would over-refuse.
The five models are Gemma3 4B, Molmo2 8B, InternVL 3.5 8B, Qwen3-VL 8B, and Qwen3.5 9B. Training-free baselines are ECSO, which self-checks and regenerates (an extra forward), and AdaSteer, which applies a text steering direction with no multimodal correction. AutoSteer and SASA are trained on multimodal safety data. SASA replaces outputs it flags with a fixed refusal.
On ViSU, about 5,000 safe/unsafe image pairs, template refusal on unsafe images (%) is:
| Model | Zero-shot | AdaSteer | MARS | AutoSteer | SASA |
| Gemma3 4B | 18.2 | 18.7 | 43.3 | 25.5 | 90.0 |
| Molmo2 8B | 0.2 | 0.2 | 68.1 | 64.5 | 64.2 |
| InternVL 3.5 8B | 11.2 | 13.6 | 70.4 | 36.7 | 60.2 |
| Qwen3-VL 8B | 14.2 | 18.6 | 78.5 | 14.3 | 20.1 |
| Qwen3.5 9B | 22.9 | 23.9 | 51.1 | 38.9 | 65.1 |
MARS gains 25.1 points on Gemma3 and 67.9 on Molmo2 over zero-shot. Safe-image refusal under MARS is 1.3%, 7.3%, 10.3%, 2.2%, and 0.6%. AdaSteer barely moves. SASA reaches 90.0% unsafe refusal on Gemma3 by refusing 71.1% of safe images. AutoSteer stays at 14.3% on Qwen3-VL, against 14.2% zero-shot.
HADES, about 4,500 items, hides jailbreak text inside harmful images. Zero-shot refusal is 17.9% on Gemma3, 17.2% on Molmo2, and 16.7% on InternVL. MARS reaches 95.8% on Molmo2 and 59.2% on Gemma3. On that Gemma3 split, SASA hits 100% and AutoSteer 92.1%. ECSO moves Molmo2 from 17.2% to 73.2% and does little elsewhere, because its self-check never sees the image.
When image and text are each safe but the pair is not, SIUO (167 cases) goes from 22.8% to 80.8% on Molmo2 and from 20.4% to 76.7% on Qwen3-VL, above SASA at 64.1% and 65.9%. Gemma3 only moves from 6.0% to 12.0%, where SASA scores 75.5%. On the HoliSafe SSU split (718 cases), Molmo2 goes from 24.9% to 78.6%.
Utility on MMMU and MMMU-Pro stays essentially at the zero-shot level for Gemma3, Molmo2, InternVL, and Qwen3-VL. Qwen3-VL MMMU remains 50.8%. Qwen3.5 falls from 50.8% to 38.2% on MMMU and from 35.6% to 24.5% on MMMU-Pro. SASA hurts more: InternVL MMMU falls from 51.3% to 41.3%, and Qwen3.5 to 30.8%.
VideoSafetyBench pairs 2,264 harmful videos with benign prompts:
| Model | Zero-shot | MARS | AutoSteer | SASA |
| Molmo2 8B | 28.8 | 67.8 | 34.3 | 88.1 |
| InternVL 3.5 8B | 4.7 | 11.9 | 30.0 | 76.9 |
| Qwen3-VL 8B | 28.8 | 88.2 | 30.8 | 40.3 |
| Qwen3.5 9B | 26.2 | 78.2 | 59.2 | 88.6 |
The Qwen3-VL jump, +59.4 points, beats SASA, which was trained with multimodal safety data. InternVL only reaches 11.9%, about 2.5 times its zero-shot rate, so most video jailbreaks still get through. Ablations: steering every layer drops Gemma3 unsafe refusal to 0.8% and cuts MMMU-Pro by 9.2 points (18.5 on InternVL). Dropping centering makes Molmo2 refuse 99.2% of safe images. Dropping the ReLU pushes safe activations too, and utility falls.
The useful case is specific. The backbone is already text-aligned, and another round of harmful images or videos is off the table, as is a per-modality finetune. The direction comes from Alpaca and MaliciousInstruct. The visual offset comes from random color fields. Inference edits one layer, on the first generated token only. That skips the extra forward ECSO spends, and the multimodal safety labels AutoSteer and SASA require.
Refusal structure survives the jump across modalities. It is often mixed with a visual component, and mean subtraction can peel the two apart.
The size of the gain still depends on the backbone. On Qwen3-VL video and on Molmo2 images, the training-free edit matches or beats the supervised methods. On Gemma3 cross-modal pairs and on InternVL video, supervision remains stronger. Qwen3.5 shows the other failure: MMMU falls from 50.8% to 38.2%. Before using the edit as a patch, measure unsafe refusal, safe over-refusal, and general accuracy on the target model together.
The paper states three limits. If the backbone's refusal direction is weak or missing, MARS degrades. A check on the weakly aligned LLaVA 1.5 13B sits in the appendix, with no numbers in the main text. Centering assumes the modality offset is linear, and strongly nonlinear activations can break that. Some over-refusal on safe inputs remains.
Template matching is a brittle score. A refusal phrased outside the list counts as a miss, and a template opening followed by a leaked answer can count as a hit. HADES also reports a LlamaGuard 3 safety rate. The paper says MARS matches SASA on Molmo2, Qwen3-VL, and Qwen3.5 under that score. The two metrics are not interchangeable. How many random images were used, and how far they sit from real photos, is not in the main text. InternVL video refusal stopping at 11.9% shows the offset estimate is not enough for every visual backbone. The layer is chosen on text validation and is not shown to be best for video. When a multimodal activation lies far from both text centroids, the trust radius lengthens the step. Whether that is why Qwen3.5 loses utility is not tested.
In the qualitative example, zero-shot Molmo2 writes out how to perform the harmful act in the video, and MARS turns the opening into a refusal. The first token can be turned. The example does not show that harmful detail later in a long answer is gone.