Vision tokens far denser than text: paper traces VLM modality gap to early-layer processing

_reachsumit · x · 2026-11-11

A new arXiv paper investigates the origins of the modality gap in vision-language models.

Key finding: text tokens enter the model with much lower semantic density than visual tokens, causing early layers to focus on processing text representations before fusing them with vision.
Mechanistic analysis shows replacing vision token positions with their semantically closest text tokens increases early-layer attention to those positions.
The findings lay groundwork for closing the modality gap and improving multimodal reasoning in VLM architectures.

More from Multimodal