Modality Gap in VLMs: Flaw or Feature?

On 08-19, @gabriberton posted a multi-part educational thread on the modality gap in vision-language models: image and text embeddings form separate clusters in shared embedding space. His conclusion: the gap may be a beneficial feature rather than a flaw, which matters for how we design and improve CLIP-style multimodal models.

Confirmed

The phenomenon appears even in randomly initialized Transformers: image embeddings tend to cluster together, so the gap partly originates from model initialization.
In dual-Transformer CLIP architectures, contrastive learning pulls positive pairs together and pushes negatives apart, but since each sample pushes away every non-paired sample, the repulsive force is often too strong, keeping the gap open in practice despite theoretical expectations.
The NeurIPS 2022 paper "Mind the Gap" systematically analyzed the phenomenon and found that closing the gap improves zero-shot classification.

Why it matters

The author argues the gap may be a feature: on the 1000-dimensional hypersphere, the gap may exist along only a single dimension, while cross-modal positive pairs (text and matching image) are actually close in most other dimensions—explaining why text-to-image retrieval works so well.
This perspective informs the design trade-off of whether one should force the gap closed in CLIP-style models.

2026-08-19 ~ 2026-08-19 · 7 related posts

Primary sources

Understanding the modality gap in VLMs and contrastive learning mechanisms — gabriberton · 2026-08-19 22:53 (UTC+08:00)
Is the modality gap a bug or a feature? Insights from high-dimensional space — gabriberton · 2026-08-19 22:53 (UTC+08:00)
NeurIPS Paper: Is the Modality Gap in Multimodal Models a Bug or Feature? — gabriberton · 2026-08-19 22:53 (UTC+08:00)

[source] Understanding the modality gap in VLMs and contrastive learning mechanisms — gabriberton · 2026-08-19
Why CLIP Training Fails to Close the Modality Gap in Multimodal Models — gabriberton · 2026-08-19
Why CLIP training fails to close the modality gap in multimodal models — gabriberton · 2026-08-19
High-dimensional reality: Why cross-modal retrieval works despite the gap — gabriberton · 2026-08-19
Why Cross-Modal Retrieval Works: Positive Samples Stay Close in Most Dimensions — gabriberton · 2026-08-19
[source] Is the modality gap a bug or a feature? Insights from high-dimensional space — gabriberton · 2026-08-19
[source] NeurIPS Paper: Is the Modality Gap in Multimodal Models a Bug or Feature? — gabriberton · 2026-08-19