ScoreCLR replaces negative samples with diffusion score networks, hits near-full-rank embeddings
canaesseth · x · 2026-11-17
Christopher Marouani, Grigory Bartosh, and Christian A. Naesseth released their first preprint, ScoreCLR (arXiv:2611.18469), a novel contrastive representation learning method.
- Core idea: estimating the repulsive force in contrastive learning typically requires large batches, has high variance, and suffers from the curse of dimensionality. ScoreCLR works in a framework maximizing mutual information between two latent views, derives an equivalent objective involving the score of the latent representation, and estimates it via denoising score matching with a lightweight network trained jointly with the encoder.
- Key advantage: a learned repulsion replaces explicit negative-sample batches, with cost linear rather than quadratic in batch size.
- Results: on CIFAR-10 and ImageNet-100 with a shared recipe, linear-probe accuracy loss vs. SimCLR is minimal, while projector outputs reach much higher effective rank — 229/256 on CIFAR-10 vs. 96 for SimCLR and 58 for LeJEPA.
The paper also proposes regularization via multiple noise levels and a Gaussian prior on the latent representation.
More from Research
- 130-page paper introduces "bottleneck pressure": work AI can't do gains value as the rest speeds up — soumitrashukla9 · 2026-11-17
- Goodfire co-founder Tom McGrath on interpretability and steering model training — adityaag · 2026-11-17
- Professor celebrates squeezing paper from 8.5 pages to the 8-page limit with AI Saiyan meme — CSProfKGD · 2026-11-17
- Stanford HAI awards $3.5M to seven teams via 2026 Hoffman-Yee Research Grants — StanfordHAI · 2026-11-17
- Motion Synthesis Models Still Demand "A person..." Prompts — Just Rewrite the Captions — andrew_n_carr · 2026-11-17
- Omar Khattab on impactful AI research: invest in projects, not papers — lateinteraction · 2026-11-17