λ-JEPA paper: projector-head anti-collapse doesn't protect backbone, SACReg boosts transfer
FrancescoLocat8 · x · 2026-10-14
An arXiv paper by Demirel et al. shows that anti-collapse objectives applied after the projection head do not necessarily prevent dimensional collapse in the backbone actually used downstream — high-rank projected representations don't guarantee a non-collapsed backbone. They introduce SACReg, a spectral anti-collapse regularizer motivated by λ-balance analysis (relative scale of weight matrices across layers), proven to prevent collapse in two-layer linear networks and shown to increase representation ranks on ImageNet100. Applied to JEPA, λ-JEPA beats LeJEPA and VISReg on ImageNet-1k classification and average linear-probe transfer across eight image datasets, and outperforms LeVJEPA and V-JEPA 2 on video self-supervised learning.
More from Research
- arXiv paper gives StopGrads theoretical grounding: convergence without an underlying loss — srush_nlp · 2026-10-23
- PEARL demo: an LLM agent that checks experiment results, revises plans, and re-tests — KevinKaichuang · 2026-10-23
- Berkeley's StyleStream 2.0 cuts streaming voice conversion latency to 520ms, 2x faster — Berkeley · 2026-10-23
- Math PhD student quits academia: AI is making math unrecognizable — pickover · 2026-10-23
- ICLR Submission Found Containing LLM's Own Meta-Comment Admitting Results Weren't Significant — _onionesque · 2026-10-23
- AKV Serving System Lets Agents Manage Their Own KV Cache, Cutting Cost 27.7% — _akhaliq · 2026-10-23