VISReg: Improving Self-Supervised Representation Collapse
机器之心 · wechat · 2026-07-14
This post introduces a new self-supervised learning project, **VISReg** (Variance-Invariance-Sketching Regularization). Co-author Haiyu Wu is from Altos Labs, and the paper was publicly endorsed and shared by Yann LeCun. ## What problem does this solve? VISReg primarily targets the issue of **representation collapse** in JEPA/self-supervised training. The authors point out that existing methods either rely on a bundle of training tricks (EMA, teacher-student, stop-grad, etc.) or, like SIGReg, suffer from vanishing gradients during collapse, making it hard to pull the model back to a normal state. ## Core Methodology VISReg breaks down regularization into three components: - **Scale constraint**: Uses a variance term to control the magnitude of each dimension, providing stable gradients even during collapse. - **Shape constraint**: Normalizes features and applies stop-gradient to the standard deviation, then uses the sliced Wasserstein distance to constrain the distribution shape. - **Centering loss**: Pulls the batch mean back to the origin. The article emphasizes that the key here is completely decoupling "scale" and "shape" to prevent them from interfering with each other. ## Results The authors conducted comparisons across 15 datasets, covering in-domain, OOD, segmentation, and generative guidance: - In **OOD generalization**, VISReg outperforms DINO, MoCoV3, I-JEPA, MAE, and data2vec across various backbone scales. - After pre-training on **ImageNet-22K**, its average OOD performance approaches that of DINOv2 trained on **10x more data**. - Following fine-tuning, it surpasses DINO and supervised pre-training on CIFAR-10, CIFAR-100, Flowers, ImageNet-1K, and Galaxy10. - It also achieved competitive or superior results in ADE20K linear segmentation and SiT image generation guidance. ## Additional Features The article notes that the computational complexity of VISReg's regularization is **O(NDK)**, which is more linear and scalable compared to VICReg's covariance term **O(ND²)**. The implementation is also quite lightweight, with the core logic requiring only about 15 lines of PyTorch code.
Related event: VISReg Targets More Stable, Generalizable SSL(2 posts)→
More from Research
- OpenForecaster uses daily news to improve language-model forecasting — Cohere_Labs · 2026-07-21
- SenseTime unveils U1 Pro and open-sources a 50M-sample vision dataset at WAIC 2026 — 机器之心 · 2026-07-21
- Baseten study finds new facts in LLM weights are fragile unless trained from many restatements — alex_verem · 2026-07-21
- Kimi K3 and Fable 5 now look much closer than the old open-vs-closed gap — FinanceYF5 · 2026-07-21
- uv-scripts/ocr returns to the top of Hugging Face datasets with a JSON model picker — vanstriendaniel · 2026-07-21
- DeepSearch-World trains web agents with 420K verifiable QA tasks — HKUST · 2026-07-21