New paper maps compute-optimal scaling laws for native multimodal pre-training

burny_tech · x · 2026-07-28

The paper studies compute-optimal scaling for native multimodal pre-training from scratch in vision-language models.
It finds that language and multimodal objectives follow different scaling laws: language allocation is relatively stable across data mixtures, while multimodal allocation is much more sensitive to composition.
A key takeaway is that text-heavy mixtures only become compute-efficient at larger scales, which shifts optimal allocation toward bigger models when the goal is better multimodal performance.
The work also reports positive transfer to downstream tasks, improving pure text spatial reasoning and robust multimodal in-context learning.

More from Multimodal