Isola's team shows a global orthogonal map aligns image and text embeddings without paired data
phillip_isola · x · 2026-10-10
Phillip Isola shares new evidence from his group (led by Dominik Schnaus, MIT/TU Munich), the paper "Shared Geometry as a Rosetta Stone," addressing pushback to the Platonic Representation Hypothesis.
Against three common criticisms — alignment may only hold in local neighborhoods, is coarse-grained/weak, or is relational and non-metric — the paper finds:
- A global map aligns image and text embeddings well, not just local neighborhoods;
- The alignment is strong enough to enable reasonable text-to-image translation without paired data;
- The map is orthogonal (plus centering and normalization), meaning the two spaces share geometry up to rotation and scale.
The method never uses pairs during training; evaluation uses FOSCTTM on 40,504 COCO validation pairs (0 = true match always nearest), plus CKA, Gromov-Wasserstein matching, and Orthogonal Procrustes as analysis tools. The framing: shared geometry acts as a Rosetta Stone — representation spaces across models and modalities are converging toward a mutually translatable structure.
Related event: MIT Team Aligns Image and Text Embeddings Without Paired Data(2 posts)→
More from Research
- Mathematician John Urschel solved a decades-old problem with GPT Astra, staying optimistic on AI research — soumitrashukla9 · 2026-10-10
- State space models bring recurrence back with linear cost and transformer-era tricks — anshulkundaje · 2026-10-10
- IROS 2026 Best Paper: humanoid robot learns tennis rallies from 5 hours of human motion data — qinzytech · 2026-10-10
- AC2 lets you train custom decision models, lifting toxicity-detection F1 from 0.482 to 0.677 — rhythmrg · 2026-10-10
- KordingLab renames Plan Your Science, keeps free AI science-planning tool for researchers — KordingLab · 2026-10-10
- QuestionFirst's AI mentor critiques research plans, not with praise but 'which result would change your mind?' — KordingLab · 2026-10-10