Study: multimodal reasoning failures stem from a 'reference gap', not perception

Guizhen Chen · hf · 2026-11-03

A new paper argues multimodal LLMs fail at multi-step visual reasoning like counting not because of perception but due to a reference gap: failure to bind language to the relevant visual entities, regions, or locations. Building on visual routines theory, the authors introduce ReGap-Bench, five paired visual reasoning tasks where scene and question are fixed while only the reference varies.

Task-matched references such as numbering improve accuracy by up to 25%; mismatched references offer little benefit or even hurt performance
Conclusion: the reference gap is a bottleneck distinct from perception; constructing task-appropriate visual references is key

Original post →

Study: multimodal reasoning failures stem from a 'reference gap', not perception

More from Multimodal