Steering Vectors Transfer Zero-Shot Between Text and Image Generation

MilaQuebec · hf · 2026-10-28

Mila/McGill with CMU, MIT and Northeastern studied whether cross-modal steering vectors exist in unified multimodal models (UMMs).

Method: find vectors that steer generation in one modality, then transfer them zero-shot to tokens of the other, going beyond representational similarity to causal control.
Findings: for diverse concepts like emotion and object size, steering vectors transfer zero-shot in both directions (text→image & image→text); even modality-unique concepts (formal vs. colloquial language) often induce consistent, interpretable effects in the other modality (e.g. more formal clothing and design).
Geometry: a clear modality gap separates text and image representations, yet transfer works — evidence of shared causal control mechanisms.

Directly relevant to interpretability and controllable generation in unified models.

More from Research