Where to Connect? Boosting MLLMs via Dynamic Gated Pathways across ALL ViT and LLM Layers
Yingying Yan, Jiaqi Tang, Wei Wei, Qianzhou Wang, Lei Zhang, Jianmin Chen, Yuyang Xia, Botong Geng, Jinjian Wu, Qifeng Chen
NeurIPS 2026)
cs.CV
2026-11-05
DGP weights all 24 ViT layers from each LLM hidden state, lifting LLaVA-1.5-7B from 66.8 to 71.0 on SQA-IMG and from 30.5 to 33.8 on MM-Vet at +5.29% parameters.
Most multimodal LLMs pass an image through a vision encoder and then give the language model one layer, usually the last or the one before it. That encoder is typically a CLIP ViT, a Transformer that cuts the image into patches and encodes them layer by layer. Edges and textures from shallow layers, and object structure from the middle, are dropped at the handoff. LLM layers sit in different semantic states and do not want the same visual granularity. One fixed connection makes every decoding depth share a single visual summary.
Later multi-layer designs still freeze the wiring. Mixing several ViT layers into one vector before the LLM finishes all adaptation before decoding starts. Sparse or hierarchical links are drawn in advance: DeepStack spreads visual tokens on a fixed schedule, and CLI gates only sampled ViT layers at chosen injection points. The full grid of language layer by vision layer never becomes a trained choice.
DGP (Dynamic Gated Pathways) lets every LLM layer, looking at its own hidden state, assign weights over all ViT layers and write the mixture back. The vision tower stays frozen. CLIP-ViT-L/14-336 has 24 layers. Each layer is summarized by its [CLS] token, the special token that pools the whole image and stands in for that layer's granularity. The summaries are mapped into the LLM hidden size and stacked as keys and values shared across language layers.
The query is a glimpse token, a learnable vector placed before the first answer token. Causal attention lets it see the entire image-text prefix, so its hidden state at a given layer is a compressed read of the context at that depth. Shared multi-head cross-attention aligns this state with the 24 summaries. A private routing MLP, hidden size 1024, turns the result into a 24-way softmax. Shared attention gives every LLM layer the same yardstick; the private network is what states that layer's preference. The weights do not vary by spatial position. Patches are mixed with one weighted sum, passed through that layer's own projector, and returned to the language hidden size.
The write-back is a gated residual. The gate is a sigmoid of a learned scalar, initialized at 0.3, so the original visual state is not overwritten at the start. Only visual tokens change. Text tokens stay as they are.
Training follows LLaVA-1.5: 558K pre-training, the mixed 665K instruction set, one epoch each, AdamW, learning rates 1e-3 and 2e-5. The LLM is updated together with the new modules. An entropy penalty with coefficient 0.01 during pre-training pushes the batch-average route off a handful of ViT layers. The coefficient is set to 0 in instruction tuning so the routes can narrow to the task.
The control is LLaVA-1.5 under the same recipe, with Vicuna as the LLM. On 7B:
| Benchmark | LLaVA-1.5-7B | DGP-7B | Delta |
| SQA-IMG | 66.8 | 71.0 | +4.2 |
| MM-Vet | 30.5 | 33.8 | +3.3 |
| VizWiz | 50.0 | 53.1 | +3.1 |
| LLaVA-Wild | 63.4 | 66.5 | +3.1 |
| GQA | 62.0 | 64.6 | +2.6 |
| POPE | 85.9 | 87.0 | +1.1 |
Where other multi-layer methods on the same backbone report a score, DGP is highest on GQA, SQA-IMG, SEED (61.5), LLaVA-Wild, MM-Vet, and MMBench-CN (60.5). Other entries are higher elsewhere. Sparse Shortcuts scores 80.0 on VQAv2 against DGP's 79.7. IGVA scores 54.3 on VizWiz against 53.1, 87.8 on POPE against 87.0, and 68.3 on SEED^I against 67.5. MMFuser scores 67.5 on English MMBench against 66.7.
At 13B, VizWiz moves from 53.6 to 57.4, MM-Vet from 35.4 to 37.5, and GQA from 63.3 to 65.5. LLaVA-Wild falls from 70.7 to 70.0; MMFuser in the same table scores 71.8. The Qwen3-VL-2B setup (SigLIP2 with Qwen3-1.7B) and LLaVA-NeXT were both retrained on 558K plus 665K, not on their original recipes. The former moves from 63.9 to 65.4 on GQA and from 85.3 to 86.9 on SQA-IMG. LLaVA-NeXT moves from 69.5 to 72.3 on SQA-IMG and from 66.5 to 68.7 on MMBench.
The ablation separates using more layers from choosing among them. A flat average of all 24 ViT layers drops GQA to 60.2, MM-Vet to 25.3, and MMBench-CN to 44.9, all below the single-layer baseline. Gated routing without the entropy term reaches 70.5 on SQA-IMG and 32.4 on MM-Vet. With the entropy term, those scores are 71.0 and 33.8.
The 7B model grows from 6759.4M to 7117.3M parameters (+5.29%). FLOPs rise 4.07%. Peak memory goes from 15.51GB to 15.49GB. The 13B model adds 4.27% parameters and 4.13% memory. Reported FLOPs fall by 0.07%.
Routes do not lock onto the last ViT layer, and they do not fall back to a flat average. Middle ViT layers are selected most often. On the 7B model, LLM layer 1 prefers shallow-to-middle features, layer 20 prefers shallow features, layer 22 prefers mid-deep features, and layer 32 prefers deep features. On one fixed image, color and small-region questions put weight earlier; a question about historical significance shifts it deeper.
The data stay put, the vision tower stays frozen, and about 5% more parameters on 7B come with more than three points on ScienceQA-IMG and MM-Vet. A flat average loses accuracy. The part that works is the choice driven by the language state.
Against LLaVA-1.5, the 7B gains hold up. Against MMFuser, IGVA, and Sparse Shortcuts, the lead sits on part of the table. Perception and hallucination benchmarks are often a tie or a loss, and 13B LLaVA-Wild goes backwards. Read this as an incremental change to how visual features are injected.
The paper does not list limitations.
The all-to-all link is between layers. The query is one [CLS] summary per ViT layer, and every patch shares that layer weight. Routing can change granularity with depth and with the question. It cannot send the background to a shallow layer and the subject to a deep layer inside one image. A single glimpse token compresses the whole prefix into the routing query.
The Qwen3-VL-2B and LLaVA-NeXT runs only show that the module can be moved. Both were retrained on LLaVA's data. GQA at 65.4 on that Qwen3 setup does not carry over to today's stronger multimodal models.
FLOPs on 13B dip after the parameter count goes up, and the paper does not explain it. That looks like measurement noise. Decoding speed is unreported. The routing study stops at heatmaps, with no link between how spread a route is and whether an example is right. The menu arithmetic and step-following cases in Figure 4 hallucinate less, and they are picked examples.