Meta/Oxford study: multimodal models need only 5% image-generation data, language training is key
rohanpaul_ai · x · 2026-08-14
A study by Meta and Oxford University finds that multimodal models may need surprisingly little image-generation data if language and visual understanding are trained together from the start. Language training already helps with vision, and learning to understand images also improves generation, but training to generate images does little for language or understanding. In 1T-token experiments, the best split was 70% language, 25% image understanding, and only 5% image generation. At 13.5B scale over 2T tokens, even with 5x fewer image-generation tokens, GenEval improved from 0.467 to 0.482, with language and understanding also improving. The study warns against adding vision too late, as models may develop 'vision laziness' and rely on language shortcuts. Paper: 'Towards Physics of Multimodal Pretraining: Knowledge Flow, Modality Synergy, Early Unification, and Recipes'.
Related event: Meta and Oxford Find 5% Image Data Unlocks Multimodal Capabilities(2 posts)→
More from Multimodal
- FrankenSim builds a physics-based virtual orchestra, piano and drum demos live — doodlestein · 2026-10-16
- vLLM officially supports Muse Spark Open: multimodal, 1M context, agentic — vllm_project · 2026-10-16
- InfoVis: 9B unified model tops open-source slide generation and editing, trained on 13M records — Jiahao Wang · 2026-10-16
- RepForcing: feature-space supervision improves few-step autoregressive video distillation — zju · 2026-10-16
- SCOPE: semantic span-aware GRPO objective improves T2I prompt rewriting — university-of-rochester · 2026-10-16
- Orbia benchmark asks: do generated videos form a consistent 3D world? — tencent · 2026-10-16