Inside the Modality Gap in VLMs: Why It Persists and When It Helps

@gabriberton posted a multi-part explainer thread on 08-19 systematically covering the "modality gap" phenomenon in vision-language models (VLMs): image and text embeddings each form their own cluster in the shared space, with a gap between them.

Confirmed

On causes: in the dual-Transformer CLIP architecture, contrastive learning pulls positive pairs together and pushes negative pairs apart, but every sample pushes away all non-paired samples—the repulsive force tends to be too strong, so the modality gap that theory predicts should close never fully closes in practice.
The phenomenon appears even in randomly initialized Transformers: image embeddings tend to cluster together, meaning the gap partly originates from model initialization.
The NeurIPS 2022 paper "Mind the Gap" analyzed this phenomenon systematically and found that closing the gap helps improve zero-shot classification performance.

Why it matters

The author argues the gap may not be a flaw but a beneficial property: hyperspherical embedding spaces are typically around 1000-dimensional, and the gap may exist along only one dimension; in most of the remaining dimensions, cross-modal positives (text and its corresponding image) are actually close to each other—explaining why cross-modal retrieval like text-to-image still works well.
This perspective offers useful insights for understanding and improving the geometry of CLIP-style multimodal models, and for the design trade-off of whether the gap should be forcibly closed.

2026-08-19 ~ 2026-08-19 · 7 related posts

Primary sources

Understanding the modality gap in VLMs and contrastive learning mechanisms — gabriberton · 2026-08-19 22:53 (UTC+08:00)
Why CLIP Training Fails to Close the Modality Gap in Multimodal Models — gabriberton · 2026-08-19 22:53 (UTC+08:00)
NeurIPS Paper: Is the Modality Gap in Multimodal Models a Bug or Feature? — gabriberton · 2026-08-19 22:53 (UTC+08:00)

[source] Understanding the modality gap in VLMs and contrastive learning mechanisms — gabriberton · 2026-08-19
[source] Why CLIP Training Fails to Close the Modality Gap in Multimodal Models — gabriberton · 2026-08-19
Why CLIP training fails to close the modality gap in multimodal models — gabriberton · 2026-08-19
High-dimensional reality: Why cross-modal retrieval works despite the gap — gabriberton · 2026-08-19
Why Cross-Modal Retrieval Works: Positive Samples Stay Close in Most Dimensions — gabriberton · 2026-08-19
Is the modality gap a bug or a feature? Insights from high-dimensional space — gabriberton · 2026-08-19
[source] NeurIPS Paper: Is the Modality Gap in Multimodal Models a Bug or Feature? — gabriberton · 2026-08-19