VISReg tackles representation collapse in self-supervised learning with sliced Wasserstein regularization

新智元 · wechat · 2026-07-28

This article explains VISReg (Variance-Invariance-Sketching Regularization), a new self-supervised learning method positioned as a response to the representation-collapse problem in JEPA-style world models.

What VISReg changes

It inherits the idea from VICReg and SIGReg, but tries to fix two weaknesses: gradient vanishing during collapse and the coupling of scale vs. shape.
VISReg splits the regularizer into three parts: a center loss, a variance/scale loss, and a shape loss based on Sliced Wasserstein Distance.
The key trick is to decouple scale and shape with stop-gradient, so the shape term does not interfere with scale control.

Why the authors care

The method aims to keep strong gradients even when representations are already collapsing, making recovery more stable than SIGReg.
The paper argues that using sliced Wasserstein on random projections can characterize the whole distribution shape, not just second-order statistics.
Implementation is lightweight; the article claims the core regularizer can be written in roughly 15 lines of PyTorch.

Reported results

Across 15 datasets, VISReg outperforms or matches seven mainstream SSL methods in many settings.
On out-of-distribution evaluation, it is consistently strong and in some settings reaches roughly 1/10 the training data of DINOv2 while matching its OOD performance.
It also shows competitive results on semantic segmentation and feature-guided generation, and is more robust on long-tailed or low-rank data.

The article frames VISReg as a practical, theoretically cleaner way to prevent collapse and improve general-purpose self-supervised representations.

Original post →

VISReg tackles representation collapse in self-supervised learning with sliced Wasserstein regularization

More from Research