Cross-modal steering vectors transfer zero-shot between text and image generation
benno_krojer · x · 2026-10-28
A new preprint led by Donghao Zeng (with bennokrojer et al.) shows steering vectors transfer zero-shot in both directions between image and text generation, offering causal evidence for a shared conceptual space in unified multimodal models.
Method
- Uses unified multimodal models (UMMs) where the same weights autoregressively generate both text and images
- Difference-in-means steering vectors extracted in text or image space, added to generated tokens; an LLM judge checks concept presence
Findings
- Across many concepts, cross-modal steering works about as well as intra-modal steering, sometimes better
- Even modality-unique concepts transfer: a vector from high-vs-low resolution images added to text makes the model describe blurry views, use shorter sentences, and express more uncertainty; colloquial "gotta" vs formal "must" also transfers into image space
- Geometry: cosine similarity between corresponding vectors correlates with steering success in some models/concepts, but no clear universal trend; the two embedding spaces remain mostly separate, with similarity arising from implicit shared structure
The authors are long-term optimistic — newer unified models steered far better than older ones — and suggest future work: diffusion-autoregression hybrids, cross-modal non-linear manifold steering, and automating the formalization of modality-unique concepts.
More from Research
- Debate set: Can we make AIs sentient — and should we? Solms vs. Humphrey — cccalum · 2026-10-29
- Samaya post-trains a 30B model to beat frontier AI on financial research at 11x lower cost — maithra_raghu · 2026-10-29
- Samaya's 30B post-trained model beats frontier on financial research at 11x lower cost — maithra_raghu · 2026-10-29
- Scaling cross-situational word learning: 2,441 hours of child-perspective video from Stanford team — RichmanRonald · 2026-10-29
- Curiosity algorithms need world models plus locomotor primitives to replicate animal behavior — aran_nayebi · 2026-10-29
- Virtual fly study finds autonomous exploration maximizes population-level state coverage — aran_nayebi · 2026-10-29