Representation loss that supervises dot products to match PMI from a generative model
serrjoa · x · 2026-10-21
Alan Baade shares a representation-learning loss design: embed two views x and y into vectors R(x), R(y), then supervise their dot product to equal the PMI between the views, computed with a jointly trained masked generative model. The loss is LPMI = ||⟨R(x),R(y)⟩ − stopgrad(log PG(y|x) − log PG(y))||², using the generative model's conditional log-probability gap as an MI estimate, with stopgrad protecting the generator from corrupting gradients.
More from Research
- ICA Lens in action: decomposing Qwen residuals reveals clean city/France features — furongh · 2026-10-21
- Into the Unknown: none of 135 submissions by 9 frontier agents passes scientific bar — wellecks · 2026-10-21
- REUF2F Benchmarks Coding Agents on Formalized Undergraduate Research Problems — wellecks · 2026-10-21
- Baker Lab and NVIDIA Unveil RFD4-Proteina, a Faster Protein Design Model That Beats Prior Methods — jueseph · 2026-10-21
- NVIDIA Kumo Forecast Pairs Pretrained Time-Series Representations with Pattern Retrieval — Hugging Face Blog · 2026-10-21
- Protein designers adopt RFD4-Proteina for Adaptyv x Anthropic competition — AllThingsApx · 2026-10-21