Towards the Physics of Multimodal Pretraining: Insights from a 13.5B MoE Model
khademinori · x · 2026-08-07
While most LLMs start with language pretraining, native multimodal training is often avoided due to instability and modality competition. This research systematically unpacks the underlying mechanics of multimodal pretraining across four key aspects:
- Knowledge Flow: Cross-modal transfer across language, understanding, and generation is highly asymmetric and concept-dependent.
- Modality Synergy: Task complexity and architectural sharing dictate whether modalities help or compete with each other.
- Early Unification: Early, simultaneous unification outperforms late or sequential training. Delayed visual integration can induce "vision laziness" in models.
- Recipes & Scale: The insights from controlled studies are used to build efficient pretraining recipes, which are validated at scale by training a 13.5B Mixture-of-Experts (MoE) model on 2T tokens.
Related event: Meta Unveils Mechanisms of Multimodal Pre-training(4 posts)→
More from Research
- AI will be superhuman at science regardless, and Europe can't opt out — teortaxesTex · 2026-10-16
- SERO augments long-horizon agentic RL with executable oracle continuations — Shanghai-AI-Laboratory · 2026-10-16
- Tsinghua and Amap propose OmniVerifier-C1, a graded verifier beating binary judges — Bowei Liu · 2026-10-16
- Alibaba open-sources MAI-Gym, an all-in-one GUI agent eval and RL training platform — Tongyi-MAI · 2026-10-16
- Peking University's SimForcing distills simulation motion priors into robot world models — PekingUniversity · 2026-10-16
- CoWorld Reasoner uses an evolving video world model as a visual reasoning engine — ModalityDance · 2026-10-16