Why CLIP Training Fails to Close the Modality Gap in Multimodal Models
gabriberton · x · 2026-08-19
- The Phenomenon: In models like CLIP using separate transformers for image and text, embeddings from different modalities cluster in distinct regions, creating a "modality gap".
- The Intuition Trap: Despite contrastive learning pulling positive pairs together and pushing negatives apart, the expected closure of this gap does not occur in practice.
- Geometric Insight: In high-dimensional hyperspheres (1000 dims), the gap may exist in only a few dimensions. While cross-modal positive samples are close along most dimensions, the overall geometric separation persists.
Related event: Modality Gap in VLMs: Flaw or Feature?(7 posts)→
More from Research
- UCSD Researchers Propose Dataset Watermarking Detectable via Black-Box API Access — yuxiangw_cs · 2026-10-16
- EmbodiedEvo tops RoboDojo as agents inherit reusable skills and lessons — Aiden_Tech_Ai · 2026-10-16
- CHI reviewer laments: papers' Discussion sections now cite nothing — IanArawjo · 2026-10-16
- Casper: Understanding 90% of neurons ≠ understanding the network — StephenLCasper · 2026-10-16
- NVIDIA Launches cuPotential in CUDA-X to Bring AI Agents to Semiconductor Materials Simulation — tomaszbednarz · 2026-10-16
- Agentic Search Systems Are Rewarded Only for Final Answers, Raising Faithfulness Concerns — burkov · 2026-10-16