LeJEPA: Provable and Scalable Self-Supervised Learning Without the Heuristics
Randall Balestriero, Yann LeCun
cs.LG, cs.AI, cs.CV, stat.ML
2025-11-12
LeJEPA regularizes JEPA embeddings toward an isotropic Gaussian with SIGReg. Dropping teacher-student and stop-gradient, a ViT-H/14 reaches 79% frozen linear probe on ImageNet-1k.
A JEPA feeds two semantically related views through the same encoder and asks one embedding to predict the other. That objective is cleaner than contrastive learning or pixel reconstruction, and it has a ready-made cheat: map every input to one vector, or into a thin subspace. Current recipes plug the hole with stop-gradient, teacher-student networks, EMA schedules, and explicit whitening. Change the hyperparameters or the backbone and the run collapses.
Balestriero and LeCun start one step earlier. If downstream tasks are unknown, which embedding distribution minimizes worst-case probe risk? Write that answer into the loss, and most of the scaffolding can go.
For a linear probe at fixed total variance, an isotropic covariance cuts both bias and variance. For radius k-NN and kernel regression under the same constraint, the unique optimum is a standard isotropic Gaussian. That is the Latent-Euclidean claim: if the probe uses Euclidean geometry, the cloud of embeddings should be a ball, not an ellipsoid.
SIGReg is the regularizer that pushes high-dimensional embeddings there. Cramér-Wold says a distribution is Gaussian once every one-dimensional projection is Gaussian. Each step samples unit directions, projects the batch, and runs an Epps-Pulley test so the empirical characteristic function matches that of N(0,1). Characteristic functions stay bounded, and so do their gradients, unlike high-order moments. Finite-moment matching is not identifying. CDF tests need sorting and fight distributed training. The characteristic-function test is what remains.
Directions are resampled every step. Sixteen fresh directions can cover more of the sphere than a few thousand frozen ones. Defaults are 1024 slices, integration on [-5, 5], and 17 trapezoid knots. The predictive term is a mean of the two global-view embeddings; every other view is pulled toward that center with squared error. The total loss is (1-λ) times prediction plus λ times SIGReg, with λ=0.05 as the starting point. No predictor, no stop-gradient, no teacher. The core is about 50 lines of PyTorch and linear in batch size.
On ImageNet-1k with a frozen backbone, the abstract reports 79% linear probe for ViT-H/14. In the body, a 0.3B ViT-Large reaches 77.1% online probe and a 0.6B ConvNeXtV2-Huge reaches 78.5% after 100 epochs. The same ViT-Large/14 recipe still scores 72.20% at batch 128 and 74.72% at batch 512. On ImageNet-10, about 50 timm models under 20M parameters, spanning eight families, land between 91.5% and 95% frozen probe. A 1.8B ViT-g trains without blowing up.
Spearman correlation between training loss and downstream accuracy is 94.52% for ViT-base on ImageNet-1k. Rescaling the loss by λ to a power near 0.4 pushes several setups toward 99%. Label-free model selection finally has a usable proxy.
In-domain pretraining is the result the paper wants remembered. Galaxy10 has about 11k galaxy images, far from natural photos.
| Method | Protocol | Galaxy10 |
| LeJEPA ResNet-34 in-domain | frozen / full FT | 78.17% / 83.28% |
| DINOv2 ViT-S/16 transfer | frozen / full FT | 67.62% / 78.34% |
| I-JEPA ViT-H/14 transfer | frozen, ImageNet-22k | 62.93% |
In few-shot transfer across eight datasets, LeJEPA ViT-L trained for 100 epochs averages 79.48% with all shots; I-JEPA ViT-H trained for 300 epochs averages 78.50%. LeJEPA wins on fine-grained sets (DTD, flowers, food); I-JEPA wins on CIFAR. On flowers102, CIFAR, and Food101, the 630M I-JEPA transfer still dominates the small in-domain models.
For anyone training visual SSL, this is an objective that moves across backbones and datasets with almost no retuning. Teacher networks and stop-gradient can come off first; SIGReg is the thing to test. When the target domain looks nothing like ImageNet, pretraining on that data with LeJEPA is more reasonable than importing DINOv2. The loss-to-probe correlation also helps pretraining loops that cannot afford labels.
This does not unseat DINOv2 on natural images at scale. Large transferred models still win there. What it shows is narrower and more useful: once the objective is specified, in-domain SSL on small data can beat generic transfer.
There is no dedicated limitations section. The abstract claims no hyperparameter schedules; the runs still use warmup and cosine decay on the learning rate. "Heuristics-free" still leaves λ, the slice count, and multi-crop views. On ImageNet-100, SWA still adds a few points on ViTs. Teacher-student is no longer required to prevent collapse, and it is not useless.
The theory is worst-case risk when the downstream task is unknown, not Bayes-optimal for a known task. Table 5 is blunt: transfer from a large natural-image model still wins on natural images; the small in-domain model wins on galaxies. The 79% ImageNet-1k number is not placed on a compute-matched table against DINOv2/v3, so "on par with SOTA" is a soft claim. The 1.8B ViT-g result is a stable loss curve, not a probe number.