New preprint: cross-modal steering vectors transfer zero-shot between text and image generation
A new arXiv preprint, "Cross-modal Steering Vectors Exist Across Both Image and Text Generation," led by Donghao Zeng with collaborators including Benno Krojer, finds that in unified multimodal models (UMMs), steering vectors learned in one modality (text generation or image generation) can transfer zero-shot to the other — in both the image↔text directions. This provides causal evidence for the hypothesis that different modalities share a common concept space.
Confirmed
- The paper comes from Mila/McGill together with CMU, MIT CSAIL, Northeastern University and others; it was led by Donghao Zeng, with Benno Krojer as a collaborator, and was also introduced by Mila's official account.
- Method: first find steering vectors that successfully steer generation in one modality, then test their zero-shot effect in the other modality.
- Demonstrated controllable concepts include emotion and object size, applicable to both text-to-image and text-to-text generation.
- Benno Krojer added an observation in the thread: cosine similarity between different steering vectors can be low, meaning multiple quite different vectors can steer the same concept well — hinting that the model may harbor multiple internal steering mechanisms.
Why it matters
- The result is a direct test of whether multimodal training generalizes knowledge across domains: if image and text share a concept space, a "concept direction" learned on text can steer image generation without extra training, and vice versa.
- Unlike correlational evidence based solely on representational similarity, steering offers causal evidence, which is valuable for understanding the internal mechanisms of unified multimodal models and for controllable cross-modal generation.
2026-10-28 ~ 2026-10-28 · 7 related posts
Primary sources
- [source] Steering Vectors Transfer Zero-Shot Between Text and Image Generation — MilaQuebec · 2026-10-28
- New preprint: cross-modal steering vectors transfer zero-shot between text and image generation — benno_krojer · 2026-10-28
- [source] Cross-modal steering vectors transfer zero-shot between text and image generation — benno_krojer · 2026-10-28
- [source] Distinct Steering Vectors With Low Cosine Similarity Can Steer the Same Concept — benno_krojer · 2026-10-28
3 near-duplicate retellings: benno_krojer · benno_krojer · benno_krojer