MIT, Stanford teams propose interpretable representation learning via PMI supervision

On October 20, MIT researcher Vincent Sitzmann posted a series of tweets detailing his team's new work on representation learning theory, aimed squarely at a problem that has long bothered him: since the rise of contrastive learning, the field has had an "alchemy" quality — lacking a formal theory describing what representations actually converge to.

Confirmed

• Sitzmann noted that methods like DINO and JEPA work well, but their training objectives are hard to explain — "black magic"
• The team's approach: use a conditional generative model to measure the mutual information between two differently masked instances of the same image, then use that mutual information estimate to directly supervise the feature dot product of the masked regions
• Models trained this way "know exactly which features they converge to," making features fully interpretable
• A key application is tokenization: when the representation learner guides diffusion processes (latent forcing or as a tokenizer), it shows excellent scaling advantages

Why it matters

• The work attempts to supply the missing convergence theory for post-contrastive representation learning, turning the objective from empirical "black magic" into an interpretable, analyzable object
• The scaling advantages in tokenization and guiding diffusion models mean this theoretical path offers not just explanatory value but potential real performance gains

2026-10-20 ~ 2026-10-20 · 8 related posts

• [source] MIT's Sitzmann: Representation learning lacks a theory of what it converges to — vincesitzmann · 2026-10-20
• Sitzmann thread (3/5): mutual information supervision for interpretable features — vincesitzmann · 2026-10-20
• Sitzmann thread (4/5): knowing exactly which features the model converges to — vincesitzmann · 2026-10-20
• [source] Sitzmann thread (5/5): favorable scaling when used for tokenization — vincesitzmann · 2026-10-20
• Sitzmann team uses mutual information to make representation learning interpretable — zetalyrae · 2026-10-20
• New diffusable representation learning objective converges to pointwise mutual information, with joint tokenizer+diffusion scaling laws — kastnerkyle · 2026-10-20
• Sitzmann: generative models are the best tools to measure information, but lack compact representations — vincesitzmann · 2026-10-20
• [source] Directly supervising representation dot products with generative-model PMI estimates hailed as a new paradigm — vincesitzmann · 2026-10-20