Just MLPs: Efficient Visual State Reconstruction for Multimodal Language Models
Jingdi lei, Junxian Li, Di Zhang, Zhanqiu Zhang, Yiwen Guo, Soujanya Poria
cs.CV, cs.AI
2026-09-28
δ-Vision uses low-rank adapters to rebuild visual KV without dropping tokens. On Qwen3-VL-4B it scores 74.4 avg, +9.1 vs best 5% pruner, at 17.90% FLOPs.
In an MLLM, a high-resolution image or a video becomes a long visual token sequence. Those tokens do two jobs. Text queries retrieve them as keys and values. They also act as live Transformer states at every layer: they form queries, take attention outputs, run through the FFN, and become a new visual state. The second job dominates cost once the sequence gets long.
The usual fix shortens the sequence. Pruning, merging, or compressing visual tokens cuts FLOPs, and it also throws away locations later layers cannot recover. Fine detail that only matters in deeper layers is gone. This paper asks a different question: keep every visual token, and stop running them through a full Transformer block at every layer.
Two measurements justify the swap. Projecting each layer's visual hidden states onto a low-rank subspace and reconstructing them keeps most downstream accuracy at ranks far below the native width. A separate d→d→d MLP per layer, trained to map the initial visual embeddings to that layer's evolved states, also matches them with high cosine similarity and low MSE. Layer-wise visual states are both low-dimensional and predictable.
δ-Vision therefore stops evolving visual tokens inside the Transformer and instead builds a visual memory per layer with a low-rank adapter. The vision encoder, projector, and LLM backbone stay frozen. Only the adapters are trained.
Each adapter is a residual correction: the output stays d-dimensional, the update lives in a subspace of rank at most r, and the up-projection starts at zero so training begins from the identity.
The memory is read-only KV. It goes through the original layer norm and the frozen K/V projections. Text queries attend to visual memory and text context together. Visual queries, visual attention outputs, and visual FFNs are skipped. Memory at layer l is discarded after use; layer l+1 gets a fresh prediction from its adapter, with no write-back from text attention. The text stream, including causal masking, is unchanged.
Training is Supervised-KD. The frozen original MLLM is the teacher, δ-Vision the student. Both see the same multimodal input and the same ground-truth answer prefix. The loss is forward KL on answer positions. An appendix ablation reports that this beats standard SFT and on-policy distillation on average, without on-policy rollouts.
Default setup: Qwen3-VL-4B-Instruct with the Embedding Adapter. Unweighted mean over nine single-image benchmarks:
| Method | Setting | Avg |
| Vanilla | 100% tokens | 80.1 |
| DivPrune | keep 5% | 65.3 |
| LLaVA-Mini | trained, keep 5% | 65.0 |
| EPIC | trained, keep 5% | 66.9 |
| δ-Vision (Emb.) | keep all | 74.4 |
| δ-Vision (Rec.) | keep all | 75.4 |
74.4 is 92.9% of the uncompressed model, 9.1 points above DivPrune at 5% retention, and slightly above the best 20%-retention pruner. The recurrent adapter adds 1.0. Per-benchmark scores for the embedding adapter: MMStar 54.4 (vanilla 64.9), RWQA 62.9 (71.2), GQA 56.7 (61.6), MMB 84.5 (87.5), POPE 87.6 (89.3), SQA 82.3 (93.3), VQAv2 78.0 (80.9). SQA drops 11.0 points, the largest single-image hit.
On Video-MME versus the uncompressed model, δ-Vision uses 17.90% of FLOPs, runs 1.30× end-to-end and 1.50× at prefill, and peaks at 12.97 GB (vanilla 12.94 GB). DivPrune sits at 16.58% FLOPs, 1.30× / 1.47×, and 10.60 GB. Same speed class as 5% pruning, no memory win, higher accuracy.
Across backbones at 5% pruning: LLaVA-1.5-7B scores 61.9 (vanilla 65.6, DivPrune 59.8); Qwen3-VL-30B-A3B scores 78.6 (82.7, 71.3); Qwen3.5-4B scores 71.6 (74.7, 62.7). All three keep about 94% to 96% of vanilla. The gap over pruning is wider on the Qwen models than on LLaVA-1.5.
On MuirBench, Video-MME, and MVBench, a single-image-only recipe averages 49.5. Adding multi-image and video data lifts that to 52.0, above the best 5% pruner at 50.2. The single-image recipe scores 40.7 on MuirBench, below DivPrune's 46.1. Keeping every token does not by itself fix cross-image alignment.
When visual-to-text influence is truncated to a low-rank subspace and passed forward, cutting all layers of Qwen3-VL-4B drops RWQA from 71.2 to 46.1 and MMStar from 64.9 to 25.6. Restoring 128 directions recovers 70.7 and 64.6. The middle 10 layers (13 to 22) hurt the most; the first 5 and last 10 barely move. Native visual attention outputs need only r95 of 38.0 to 103.5 on this model.
This adds a second efficiency axis for MLLM inference: compress how visual states are built, leave the sequence length alone. Spatial evidence stays in place, which is the setting where pruning tends to drop fine-grained items. Adapters are small and the backbone is frozen, so the extra cost is a short distillation, not a full retrain.
It does not beat 5% pruning on speed, and peak memory is essentially the vanilla number because every visual KV slot remains. Treat it as an accuracy-first alternative that accepts an unchanged KV cache, not as a strict upgrade over pruning. An appendix says it can be stacked with DART and DivPrune; those numbers are not in the main tables.
If FastV, VisionZip, or DivPrune already cost too much accuracy on detail-heavy questions, this is the path to try: keep the evidence, fit a low-rank correction to whatever each layer needs to read.
There is no Limitations section in the main text. The tables still show the gaps.
Accuracy stays below vanilla. On Qwen3-VL-4B the two adapters land at 74.4 and 75.4 against 80.1, a 5.7 to 7.1 point hole. SQA falls from 93.3 to 82.3, so science QA seems to need more of the original visual evolution than a low-rank residual provides.
Most headline baselines are training-free pruners, while δ-Vision needs a distillation stage, so the comparison is not symmetric. It does beat the trained compressors LLaVA-Mini and EPIC by 9.4 and 7.5 points, but the main table also omits methods such as EE-MLLM that drop visual self-attention while keeping the original layer compute.
Visual memory is read-only. Nothing in a layer writes text information back into the visual state. Tasks that need joint visual-text rewriting are untested. The single-image recipe already loses to pruning on MuirBench.
The predictability study uses a full-width d→d→d MLP per layer; the deployed module is a low-rank bottleneck. Those are different objects, and the title Just MLPs invites mixing them. The bottleneck rank r and the training mix live in the appendix; the main text does not state r.
Peak memory is 12.97 GB, slightly above vanilla at 12.94 GB. If memory is the constraint, this paper does not help. End-to-end speedup is 1.30× against 1.50× at prefill, because decode still reads the full visual KV. Long generations will see even less of that prefill gain.