Physics of Multimodal Pretraining: Synergy and Competition Between Vision and Language

heghbalz · x · 2026-08-06

This arXiv paper explores the design space and fundamental mechanisms of natively unified multimodal pretraining. Through controlled experiments, the authors present four key findings:

Knowledge Flow: Disentangles how language, visual understanding, and visual generation transfer knowledge across modalities, revealing distinct patterns of influence and asymmetry.
Synergy vs. Competition: Data "complexity" determines whether modalities synergize or compete. Shared attention and normalization layers, combined with modality-specific feed-forward networks, promote synergy.
Early Unification: Jointly training modalities from the very early stages is more effective than late alignment or sequential training, though it uncovers a "vision laziness" phenomenon if delayed.
Architectural Recipes: Discusses the impact of different visual tokenizer designs and provides practical training recipes.

Original post →

Physics of Multimodal Pretraining: Synergy and Competition Between Vision and Language

More from Models