Meta Unpacks Multimodal Pretraining: Strong Generation with 5% Compute

facebook · hf · 2026-08-06

Meta's new paper systematically explores the mechanisms of unified multimodal pretraining, yielding four key insights:

Knowledge Flow: Disentangles cross-modal influence and asymmetry among language, visual understanding, and generation.
Synergy vs. Competition: Data complexity dictates synergy; shared attention and modality-specific feed-forward layers promote it.
Early Unification: Jointly training modalities from early stages outperforms late alignment, avoiding 'vision laziness' where models rely on language priors.
Recipes: Derives efficient pretraining recipes achieving strong generative performance using only 5% of compute.

Findings are validated at scale on 13.5B MoE models trained on 2T tokens.

More from Multimodal