Why block masks work in JEPA: 151 pretraining runs show unrecoverable content is key
udmrzn · x · 2026-10-01
An arXiv paper gives a measurement account of JEPA's masking sensitivity: block masks work not because of their shape but because they hide coarse-scale content that low-level interpolation cannot recover. Across 151 pretraining runs on ImageNet-100, strip masks matching blocks in area and contiguity hit 40.3% linear top-1 vs 64.3% for blocks; within one geometry family placements leaving the least unrecoverable content lose 6.5 points; pixel targets span 7 points where latent targets span 25; and against a frozen target the 19-point gap between random and block masks closes to 1.5. On UCF101, masking ratio decides whether content unrecoverability or context reachability binds.
More from Research
- New lower bounds for kissing numbers: τ₁₉≥12268 and τ₂₁≥30761, open-sourced — felpix_ · 2026-10-01
- New paper: LLMs can get answers right while their chain-of-thought traces are invalid — rao2z · 2026-10-01
- Gemini 3.8 Flash (high) hits 84.8% on WeirdML v2, first Flash to beat Gemini 3.1 Pro — teortaxesTex · 2026-10-01
- Researchers steal frontier models' hidden reasoning by replaying encrypted chain-of-thought traces — maksym_andr · 2026-10-01
- Thinking in Geometric Terms: What ReLU, LayerNorm, LoRA and VQ Do to the Data Space — techNmak · 2026-10-01
- First superhuman Stratego AI unveiled in Nature paper using RL and test-time compute — zicokolter · 2026-10-01