Tencent Hunyuan maps scaling laws for native multimodal pretraining from scratch

Tencent-Hunyuan · hf · 2026-07-27

Tencent Hunyuan studies how native multimodal pre-training from scratch scales under a fixed compute budget.
The paper finds that minimal loss follows a predictable compute law, while compute-optimal model size and token count follow power laws.
Language and multimodal objectives behave differently: the language allocation law is mostly invariant to data composition, but the multimodal allocation law is highly sensitive to it.
Text-heavy mixtures only become compute-efficient at larger model scales, pushing the optimal allocation toward more model capacity.
The authors also derive an efficiency frontier that specifies model size, token count, and data mixture, and report positive cross-modal transfer to pure-text spatial reasoning and multimodal in-context learning.

More from Multimodal