λ-JEPA adds spectral anti-collapse regularization, beating LeJEPA and VISReg

ClementineDomi6 · x · 2026-09-30

Researchers from the Locatello group released λ-JEPA: Spectral Anti-Collapse Regularization for Self-Supervised Learning.

Problem: In joint-embedding SSL, invariance and anti-collapse objectives act after the projector head while downstream tasks use pre-projector backbone features; the authors show this mismatch does not prevent dimensional collapse in the backbone, which can retain low effective rank and limit transfer.
Method: They propose SACReg, a spectral anti-collapse regularizer motivated by λ-balance analysis of relative weight-matrix scales. In a two-layer linear network they prove λ-balance prevents collapse and that SACReg induces it; empirically it raises representation ranks on ImageNet100.
Results: Applied to JEPA, λ-JEPA outperforms LeJEPA and VISReg on ImageNet-1k classification and average linear-probe transfer across eight image datasets, and beats LeVJEPA and V-JEPA 2 on video SSL benchmarks.

More from Research