Theoretical Analysis on Why Contrastive Learning Works
MIT · hf · 2026-07-15
This work seeks to answer: why can contrastive learning with simple images and augmentations learn useful representations for downstream tasks? The authors analytically derive the **optimal representation** of the contrastive loss on a class of basic augmentations and image datasets satisfying stationary statistics. The results show: - Under certain augmentation settings, the optimal solution can be implemented by a CNN with the following structure: - The first layer consists of **sinusoidal filters** - Followed by pointwise non-linearity - Global average pooling - A final linear layer performing **partial whitening** - For more complex augmentations, the optimal weights still take a sinusoidal form. - The frequency and weights of the sinusoids can be computed using a simple **waterfilling** algorithm based on the expected power spectrum of the dataset. Experiments across various image datasets and augmentation settings validate this analysis: networks trained with SGD indeed learn sinusoidal-style filters in the first layer and exhibit partial whitening behavior.
Related event: Theoretical Analysis on the Effectiveness of Contrastive Learning(2 posts)→
More from Research
- GigaChat Audio targets long-form audio grounding with timestamps across 120-minute inputs — ai-sage · 2026-07-21
- Paper models Transformer components as stochastic geometry and tests five architectures — Zhihua Liang · 2026-07-21
- LTX 2.3 LoRA demo changes a video’s camera angle — CQDSN · 2026-07-21
- OpenForecaster uses daily news to improve language-model forecasting — Cohere_Labs · 2026-07-21
- Baseten study finds new facts in LLM weights are fragile unless trained from many restatements — alex_verem · 2026-07-21
- uv-scripts/ocr returns to the top of Hugging Face datasets with a JSON model picker — vanstriendaniel · 2026-07-21