SIGReg regularizer in LeJEPA and LeWM reveals hidden contrastive pairwise repulsion in JEPAs
burny_tech · x · 2026-09-29
- Common wisdom holds that JEPAs are inherently non-contrastive.
- But recent models like LeJEPA and LeWM use the SIGReg regularizer, which approximates a sliced MMD; its expansion reveals pairwise repulsion between samples within a batch.
- So these JEPA variants actually contain a contrastive-style repulsion mechanism, making the "JEPA is non-contrastive" claim strictly inaccurate.
More from Research
- Late-layer neurons in Qwen act like on-off switches, unlike Olmo — Sauers_ · 2026-09-29
- PyroAdapt lifts California wildfire prediction, boosting extreme-fire recall by 39.7 points — UIUC-CS · 2026-09-29
- KAIST proposes EAPO: entropy-guided credit assignment reinforcing surprising success in RLVR — kaist-ai · 2026-09-29
- Perturbed public documents synthesize training data, lifting Qwen 35B to trillion-param level — AllSpark-Research · 2026-09-29
- Skill2Env synthesizes executable environments from skills for agent post-training — AllSpark-Research · 2026-09-29
- Bigger than CRISPR? Nature guides readers through the latest genome editors — LexiLove · 2026-09-29