LeVJEPA: Efficient & Scalable Video Pretraining without the Heuristics
Lukas Kuhn, Lucas Maes, Giuseppe Serra, Quentin Le Lidec, Yann LeCun, Randall Balestriero, Florian Buettner
cs.CV, cs.AI
2026-08-28
LeVJEPA trains one video encoder with SIGReg, no teacher. Matched epochs track V-JEPA 2 at 5.6-20.8x less compute; matched FLOPs lead ImageNet by 7.6 points.
Video is a richer pretraining substrate than still images and a more expensive one. Leading self-supervised recipes pay for that cost twice. Joint-embedding methods such as V-JEPA and V-JEPA 2 keep a second EMA encoder, a stop-gradient, and a capacity-limited predictor so the representation cannot collapse. Pixel reconstructors such as VideoMAE avoid collapse by imputing masked patches, which forces structured tube masks so the task is not trivial. In both lines, a large share of each step exists only to keep training stable: a teacher forward pass over the full clip, a predictor over masked queries.
LeJEPA already offered a different collapse control on images: SIGReg pushes embeddings toward an isotropic Gaussian and rules collapse out with a guarantee, so the architecture shrinks to one network. This paper is the first video encoder trained under that objective. Two consequences follow. If the heuristics can go, pretraining cost is only the tokens the encoder sees. If the two branches no longer need to be asymmetric, the encoder can be causal in time.
A 16-frame clip yields one global view and V local views (default V=4) that share the same temporal window. Local views are spatially cropped and photometrically augmented. One encoder Eθ processes every view. A small projector maps the [cls] token into K dimensions, where the loss lives. The projector exists because the last layer's LayerNorm pins [cls] to a sphere, a geometry that fights SIGReg; it is dropped after pretraining and downstream work uses encoder features.
The loss has two terms. An MSE pulls each local [cls] toward the global [cls], with gradients flowing through both sides and no stop-gradient. SIGReg then projects the batch onto random directions and penalizes, with the Epps–Pulley statistic, any projection that fails a standard-normal test. Collapse, which has zero variance along some axis, is then excluded by construction. The only hyperparameter is λ, fixed at the published default 0.02 in every run.
During pretraining, 95% of patch tokens in each view are dropped uniformly at random. Step cost tracks the tokens the encoder actually sees, so dropping is both a FLOP cut and an augmentation: the clip embedding has to be recoverable from a sparse, randomly located sample. Attention is block-causal by default, bidirectional inside a frame and causal across frames; [cls] attends to everyone and is never attended to. Factorized 3D RoPE lets one set of weights handle both view resolutions.
Default data is a class-balanced 20% subsample of K710 (Kinetics-400/600/700 with validation overlap removed). Evaluation is frozen: attentive probing on ImageNet-1K and Something-Something-v2, linear probing on mean-pooled tokens for Kinetics-400, where an attentive probe would be disproportionate.
Baselines were retrained on the same 20% K710 split, 240 epochs, effective batch 3,072.
At matched epochs, LeVJEPA matches or beats V-JEPA 2 across ViT-S/B/L at 5.6× to 20.8× less total pretraining compute. On ViT-B the accuracy gap is under one point while compute is 4.8 versus 36.4 ExaFLOPs. On ViT-L, LeVJEPA is 1.9 points ahead at 5.6× lower cost, and that ViT-L still uses less than half the compute of V-JEPA 2's ViT-S.
At matched total FLOPs the cheaper step buys a longer schedule (1,085 epochs, V=10):
| Method | IN1K | SSv2 | K400 |
| VideoMAEv2 | 53.4 | 43.6 | 37.4 |
| V-JEPA 2 | 51.6 | 42.5 | 40.7 |
| LeVJEPA | 61.0 | 40.4 | 44.6 |
ImageNet leads the strongest video baseline by 7.6 points and Kinetics-400 is highest; Something-Something-v2 trails VideoMAEv2 by 3.2 points. Against DINOv2 trained on frames of the same videos at equal FLOPs, ImageNet is 50.7 versus 53.8 (a 3.1-point gap) while SSv2 is 30.4 versus 16.9, nearly double on motion.
Token dropping helps ImageNet monotonically: 33.9% with every token kept, 47.6% at ρ=0.95. A tube pattern that keeps the same spatial sites in every frame drops ImageNet from 50.7% to 39.6%. Per-frame tokens (τ=1) beat two-frame input aggregation (τ=2): 50.7 versus 47.4 on ImageNet, 30.4 versus 28.8 on SSv2. Block-causal attention is 51.2% against 50.7% bidirectional, so causality is effectively free.
Scaled data (full K710, SSv2, Walking Tours, PE Video) for 100 epochs of ViT-L/16 reaches 69.5% ImageNet and 55.0% SSv2 under frozen attentive probing, 9.5 ImageNet points above the same size on the 20% K710 split. On a single RTX 5080 (16 GB), a ViT-Tiny trained 12 hours on eight unlabeled Walking Tours videos moves ImageNet from 8.9% at init to 25.2%. The same card fits batch 128 in under 8 GB; a V-JEPA encoder of the same size saturates at batch 28.
Only [cls] is supervised, yet patch-token PCA still separates objects from background. Public V-JEPA 2 does not show that structure; V-JEPA 2.1 gets it from an extra patch-level loss.
A chunk of video pretraining cost was the teacher–predictor scaffold, not the video. Once that scaffold is gone, 95% random dropping is an augmentation rather than an approximation, and mask geometry, temporal aggregation, and attention causality become knobs you set for the downstream job. For streaming perception or an autoregressive world model, block-causal attention has a practical payoff: a new frame does not require re-encoding the past.
The image comparison is closer to a stance than a leaderboard flex. At equal compute on frames from the same videos, appearance trails DINOv2 by about three points and motion nearly doubles. Video is no longer an add-on you pay for only when you need actions. The gain is large. The idea is transferring LeJEPA to video and showing that, with the heuristics removed, the efficiency dividend is big enough to matter.
Motion is the remaining weak axis. Aggressive dropping hurts SSv2 on short schedules; longer training recovers most of it inside the same compute, but a dropping scheme that keeps temporal correspondences at high sparsity is future work. Controlled comparisons stop at 20% of K710 and ViT-L. How SIGReg behaves at very large batch and model scale is unknown, and there is no internet-scale bake-off against V-JEPA 2. Dense tasks such as segmentation and tracking were not evaluated, so it is unclear whether the unsupervised patch structure is enough.
In the FLOP-matched table LeVJEPA also received more epochs and more local views. Part of the win is "the cheaper step buys more iterates," which is a fair property of the method and should not be read as a same-step miracle. Kinetics-400 uses a weaker linear probe, so those numbers are conservative and comparable across methods, but they should not be mixed with the attentive-probe scores on ImageNet and SSv2. The 25.2% consumer-GPU ImageNet run shows the recipe fits on a desktop; it is not a usable representation.