DGP: letting every LLM layer route to all ViT layers boosts MLLMs across 11 benchmarks

机器之心 · wechat · 2026-11-10

NPU/HKUST researchers propose DGP (Dynamic Gated Pathways): instead of feeding one fixed ViT layer to the LLM, every LLM layer can access all ViT layers, dynamically weighting them per image-question context — turning "which vision layer connects to which language layer" into a learnable routing problem.

Mechanism: a learnable glimpse token queries per-layer, ViT [CLS] tokens serve as keys/values; a shared cross-attention plus per-layer routing MLPs produce Softmax weights, fused per-patch-position and injected via gated residual; entropy regularization during pretraining prevents premature routing collapse
Results: all 11 metrics improve on LLaVA-1.5-7B (SQA-I +4.2, MM-Vet +3.3); consistent gains on Qwen3-VL-2B and LLaVA-NeXT
Key ablation: uniformly averaging all 24 ViT layers hurts (MM-Vet 30.5→25.3), showing gains come from learned routing
Visualization: texture questions route to shallow layers, semantic questions to deep ones; same image, different question → different routing
Cost: +5.29% params, +4.07% FLOPs on 7B, peak memory roughly unchanged

Paper: arxiv.org/abs/2611.06844; code: github.com/YingyingYan1/DGP

Original post →

DGP: letting every LLM layer route to all ViT layers boosts MLLMs across 11 benchmarks

More from Models