V-JEPA-LM: Video-Language Models That Read Dynamics Beyond Static, Text-Aligned Representations
Keuntae Kim, Dongjun Kim, Yong Suk Choi
cs.CV, cs.AI
2026-11-13
Captions and QA leave V-JEPA's motion cues unread; counterfactual pixel-transform data fixes that, giving a 4B model 72.1 average over 11 benchmarks, 11 points above the best open 4B.
V-JEPA is a self-supervised video encoder trained without captions. Its features carry how a scene changes: direction, speed, order, state. But when it gets wired to a language model, the usual image-text VLM recipe takes over: alignment, pretraining and instruction stages on captions and QA. Meta's V-JEPA 2 video LLM used 88.5M samples and 288 tokens per frame.
The trouble is what that supervision asks for. A caption or question names a few attributes of a clip, and one frame often answers it. The model learns to look up what the text mentions, not to read the clip. The paper's clearest number: on held-out speed-ratio questions, a linear probe reads the answer from frozen V-JEPA features at 86.7% and still gets 83.5% from the bridge output, yet the VLM answers at 33.7%, which is chance for three options. The information arrives and nobody needs it.
Two steps: measure where the change signal lives, then build data that requires it.
| Model | Metric | Score |
| V-JEPA-LM 4B | mean of 12 scores (11 benchmarks) | 72.14 |
| Best open 4B (Molmo2-4B) | same | 61.08 |
| Best open 8B (PLM-8B) | same | 64.79 |
| V-JEPA-LM 2B | same | 66.02 vs 52.45 for best open 2B |
| V-JEPA-LM 8B (Llama 3.1) | same | 70.51 |
The models use 2 to 38 times fewer training samples than the open models that report counts. On TOMATO the 4B model scores 53.5 against at most 36.5 for open 4B models. Against Meta's unreleased V-JEPA video LLMs on the six benchmarks both report, the 8B models lead by 6.7 and 4.9 points with about a twentieth of the data.
In the controlled 2B study, V-JEPA ends at 66.02 with counterfactual rows and 61.53 without; SigLIP ends at 62.13 and 58.52. On speed questions the rows add 32.8 points for V-JEPA but 11.4 for SigLIP. On held-out sets (unseen speed ratios and so on) V-JEPA-LM scores 94.5 to 96.3 and open models 43.5 to 55.5. Replacing the rows with a single arrow-of-time question gives 61.17, no better than captions and QA alone (61.53), so the gain comes from the construction itself.
The probe table agrees: with the rows, the same speed items go from 33.7% to 92.8% answer accuracy.
The paper's takeaway is concrete: the encoder decides what can be read, the supervision decides what is read. Direction, counting and fine-grained actions come mostly from the encoder. Speed, camera motion and state change only appear with counterfactual supervision.
For anyone building video understanding, this is a cheap recipe. The data comes from scripts, not annotators or an LLM, and a new video collection plugs in easily. One stage, under 5M samples and 32 tokens per frame also cuts inference prefill FLOPs by 2 to 18 times against same-size open models.
It is also a counterexample to the scale-the-data route. The gains are concentrated in temporal-dynamics tasks, though; general knowledge scores are flat.
The authors say the encoder predicts video rather than reading text, so text recognition and single-image tasks are out of scope, and hour-long video was too costly to train on.
Several things remain unverified: