V-JEPA-LM: counterfactual supervision teaches LMs to read video dynamics with 20x less data
Keuntae Kim · hf · 2026-11-16
V-JEPA-LM shows that counterfactual supervision over pixel transformations — the same clip played forward vs. backward, where only the change determines the answer — lets language models read motion dynamics from a frozen V-JEPA predictive video encoder instead of static, text-aligned features.
- Motivation: caption/QA supervision is answerable from one frame; per-frame encoders like SigLIP carry near-zero dynamics in-token (consistency κ 0.0016) while V-JEPA's spatiotemporal tubes encode it (κ 0.167)
- Method: frozen encoder, single training stage, pair-gradient supervision from time-reversed clips
- Efficiency: training stages 3→1, samples 88.5M→4.53M, visual tokens per frame 288→32
- Results: 4B-class accuracy rises from 61.1 to 72.1 across 12 benchmarks, beating InternVL3.5-4B and PLM-3B with 1/20 the training data
More from Multimodal
- No underwater IQA metric matches human perception: 73 metrics tested, best kappa just 0.23 — DelhiTechnologicalUniversity · 2026-11-16
- Midjourney sref codes render knit and embroidered Adidas-style retro sportswear — sergeantsref · 2026-11-16
- TetraCoder: adaptive tetrahedral encoding beats Instant-NGP PSNR with fewer parameters — ssh4net · 2026-11-16
- Open-source ComfyUI node enhances MiniMax H3 image conditioning for better texture and detail — Away_Exam_4586 · 2026-11-16
- Combining LTX Audio Nodes with MiniMax H3 for Lip-Synced Video in ComfyUI — CQDSN · 2026-11-16
- One character, two styles: AI generates matching 2D illustration and 3D render — azed_ai · 2026-11-16