Cross-modal Steering Vectors Transfer Zero-shot Between Image and Text Generation, New Paper Shows
benno_krojer · x · 2026-10-28
A new arXiv paper from Mila/McGill, CMU, and MIT CSAIL researchers shows that steering vectors found in unified multimodal models (UMMs) transfer zero-shot across modalities: vectors that steer one concept in text generation also steer the same concept in image generation, and vice versa.
Key points:
- Unlike correlational methods (e.g. representational similarity), steering is a causal intervention, indicating mechanisms are unified across modalities.
- For modality-unique concepts (e.g. formal vs. colloquial language), steering still induces consistent, interpretable effects in the other modality (e.g. formal clothing/design).
- The transfer has a geometric signature despite the modality gap between text and image representations.
- The first author, Donghao Zeng, did this as his first research experience; future ideas include diffusion-autoregression hybrids and formalizing modality-unique concepts.
Paper, demo, and code are all linked.
More from Research
- "3SUM was solved 5 days ago, feels like 10 years": the breakthrough everyone keeps reposting — gleech · 2026-10-28
- Gym-Anything-RT: best frontier model solves only 9% of real-time tasks vs 45% for humans — wellecks · 2026-10-28
- Blind test of 152 listeners: Suno matches human composer, but almost no one can tell which is AI — teropa · 2026-10-28
- Adding a "Behind the Scenes" appendix to papers to preserve research intuition — benno_krojer · 2026-10-28
- EMNLP 2026 panel: How to measure the positive social impact of AI and NLP — ZhijingJin · 2026-10-28
- Baidu's quaternion-based oHC residual connections cut BPB by 6.2σ but only delay stream homogenization — teortaxesTex · 2026-10-28