Sitzmann team uses mutual information to make representation learning interpretable
zetalyrae · x · 2026-10-20
Vincent Sitzmann shared a thread on his team's new representation learning work, arguing the field has been alchemy since contrastive learning: objectives like DINO and JEPA work but lack a theory of what representations converge to.
- Use conditional generative models to measure mutual information between two mask instantiations of an image
- Use that estimate to directly supervise the dot product of the features of masked regions
- Result: a model where you know exactly which features it converges to
Key application is tokenization: guiding the diffusion process via latent forcing or as a tokenizer, the representation learner shows very favorable scaling behavior. Collaborators include Alan Baade, Gordon Wetzstein, Eric Chan, plus Fei-Fei Li, Jon Barron and Kyle Sargent.
More from Research
- Turing Post's Must-Read Papers of the Week: Cortex, Librarian Agents, Mosaic and 12 More — TheTuringPost · 2026-10-20
- MTSQL-R1 Targets the Short-Horizon Habits That Break Multi-Turn Text-to-SQL — burkov · 2026-10-20
- Robotics take: IK guarantees no precision edge, hardware—not the solver—is the bottleneck — ChongZzZhang · 2026-10-20
- MPC vs RL for robot tracking: the precision bottleneck is hardware, not the solver — ChongZzZhang · 2026-10-20
- Developer Spends 1.6M Tokens Having Codex Benchmark MPC vs RL for Arm Tracking — ChongZzZhang · 2026-10-20
- Terminal Bench 4 audit: 44 of 65 tasks cheatable, Sonnet 5.5 hits 89.2% after fixes — scaling01 · 2026-10-20