V-JEPA-LM: Counterfactual Video Data Lets a 4B Model Score 72.1, Beating Open 8B Models

V-JEPA-LM: Video-Language Models That Read Dynamics Beyond Static, Text-Aligned Representations

Keuntae Kim, Dongjun Kim, Yong Suk Choi

cs.CV, cs.AI

2026-11-13

Captions and QA leave V-JEPA's motion cues unread; counterfactual pixel-transform data fixes that, giving a 4B model 72.1 average over 11 benchmarks, 11 points above the best open 4B.

What problem this solves

V-JEPA is a self-supervised video encoder trained without captions. Its features carry how a scene changes: direction, speed, order, state. But when it gets wired to a language model, the usual image-text VLM recipe takes over: alignment, pretraining and instruction stages on captions and QA. Meta's V-JEPA 2 video LLM used 88.5M samples and 288 tokens per frame.

The trouble is what that supervision asks for. A caption or question names a few attributes of a clip, and one frame often answers it. The model learns to look up what the text mentions, not to read the clip. The paper's clearest number: on held-out speed-ratio questions, a linear probe reads the answer from frozen V-JEPA features at 86.7% and still gets 83.5% from the bridge output, yet the VLM answers at 33.7%, which is chance for three options. The information arrives and nobody needs it.

Method

Two steps: measure where the change signal lives, then build data that requires it.

Results

ModelMetricScore
V-JEPA-LM 4Bmean of 12 scores (11 benchmarks)72.14
Best open 4B (Molmo2-4B)same61.08
Best open 8B (PLM-8B)same64.79
V-JEPA-LM 2Bsame66.02 vs 52.45 for best open 2B
V-JEPA-LM 8B (Llama 3.1)same70.51

The models use 2 to 38 times fewer training samples than the open models that report counts. On TOMATO the 4B model scores 53.5 against at most 36.5 for open 4B models. Against Meta's unreleased V-JEPA video LLMs on the six benchmarks both report, the 8B models lead by 6.7 and 4.9 points with about a twentieth of the data.

In the controlled 2B study, V-JEPA ends at 66.02 with counterfactual rows and 61.53 without; SigLIP ends at 62.13 and 58.52. On speed questions the rows add 32.8 points for V-JEPA but 11.4 for SigLIP. On held-out sets (unseen speed ratios and so on) V-JEPA-LM scores 94.5 to 96.3 and open models 43.5 to 55.5. Replacing the rows with a single arrow-of-time question gives 61.17, no better than captions and QA alone (61.53), so the gain comes from the construction itself.

The probe table agrees: with the rows, the same speed items go from 33.7% to 92.8% answer accuracy.

Why it matters

The paper's takeaway is concrete: the encoder decides what can be read, the supervision decides what is read. Direction, counting and fine-grained actions come mostly from the encoder. Speed, camera motion and state change only appear with counterfactual supervision.

For anyone building video understanding, this is a cheap recipe. The data comes from scripts, not annotators or an LLM, and a new video collection plugs in easily. One stage, under 5M samples and 32 tokens per frame also cuts inference prefill FLOPs by 2 to 18 times against same-size open models.

It is also a counterexample to the scale-the-data route. The gains are concentrated in temporal-dynamics tasks, though; general knowledge scores are flat.

Limitations

The authors say the encoder predicts video rather than reading text, so text recognition and single-image tasks are out of scope, and hour-long video was too costly to train on.

Several things remain unverified:

Terms

Source

Related papers

All paper explainers