Meta AI: Mid-training LLM on raw video lifts multimodal scores without text loss

kastnerkyle · x · 2026-10-10

A Meta AI paper (arXiv:2610.11019) tests whether raw, caption-free web video can serve as mid-training data for a pretrained LLM.

Method: video frames are encoded into continuous visual tokens; Qwen3-1.7B is mid-trained via next-visual-token prediction on YT-Temporal-1B, fully self-supervised with no text loss
Results: +2.9 points average across 4 video benchmarks and +5.1 across 10 image benchmarks vs. no mid-training; text performance preserved (48.9 vs 48.0 over 14 text benchmarks)
Training dynamics: image/video gains emerge within the first 30% of training then plateau (within 0.5 points)
Predicting captions underperforms next-visual-token prediction, avoiding captioning compute and labeling noise

More from Multimodal