VisCo: Leveraging Large Language Models as Intrinsic Encoders for Visual Token Compression
Yupeng Zheng, Kai Zou, Bin Liu, Nenghai Yu
cs.CV
2026-07-14
VisCo turns a VLM into its own compressor; the decoder reuses the layer-wise KV of a few memory tokens, and at one visual token it still holds 85.3% of performance, beating all baselines.
A VLM (vision-language model) chops an image into hundreds of visual tokens and feeds them to its language backbone, so self-attention cost and KV-cache memory balloon with token count. High-resolution images make it worse, and latency and memory become the deployment bottleneck. Visual token compression means representing the same image with fewer tokens.
The two existing families both have hard limits. Training-free methods (FastV and SparseVLM prune by attention scores; DART and VisionZip merge by feature similarity) hold up under mild compression but collapse when you push the ratio, because they drop fine-grained detail and global semantics, and attention shift can induce hallucination. Training-based methods either bolt on an external compression module (Q-Former, PruMerge) that forces the VLM to re-adapt at high retraining cost and damages the pretrained priors, or, like VoCo-LLaMA, redo alignment and instruction tuning: strong but expensive and not plug-and-play. Push the ratio hard and the free methods break while the trained methods cost too much.
The authors' take: compression is fundamentally an encoding problem, and the pretrained VLM already encodes well, but no one uses that capacity. The fix is to let the VLM compress itself with only lightweight adaptation, rather than bolting on a module that forces it to change.
VisCo turns the pretrained VLM into a parameter-sharing autoencoder: the encoder is the VLM and the decoder is the same VLM. The compressed representation is a small set of learnable memory tokens. Three steps.
Sequence construction. An image passes through the vision encoder and projector to give Nv visual tokens Xv. VisCo appends Nm learnable memory tokens Xm (Nm much smaller than Nv) to form [Xv; Xm]. Because VisCo keeps the VLM's native causal masking, each memory token, sitting after the visual tokens, can attend to all preceding visual tokens with no custom interaction rule. This step relies purely on the VLM's own attention prior.
Layer-wise aggregation. Transformers process information hierarchically: shallow layers capture texture and local detail, deep layers capture abstract semantics. VisCo does not take only the encoder's final output (that would flatten the hierarchy); it stores the K and V of the memory tokens at every layer into a memory bank, discards the encoder's final hidden states, and relies on these layer-wise KV pairs alone at decode time. This preserves both shallow detail and deep semantics.
Hierarchical prefix decoding. The decoder is the frozen pretrained VLM (LoRA removed). VisCo pours the layer-wise KV from the memory bank directly into the decoder's KV cache and then autoregresses on top of it. Because encoder and decoder share parameters, their representational and attention priors match, so the decoder reuses the KV with no projection, unlike Prefix-Tuning which adds per-layer MLPs. Only the memory tokens and LoRA (rank 64) are trained, fine-tuned for one epoch on a 10% subset of LLaVA-665K.
Evaluation spans 3 VLM backbones (LLaVA-1.5-7B, Qwen2-VL-2B, Qwen2-VL-7B) and 6 benchmarks (GQA, MMB, MMB-CN, MME, POPE, MMVet). Baselines are FastV, SparseVLM, PruMerge+, DivPrune, VisPruner, and a fine-tuned VisionZip, plus VoCo-LLaMA and MatryoshkaQuery as heavier references.
The headline finding: the harder you compress, the bigger VisCo's lead. On LLaVA-1.5-7B, going from 576 to 32 tokens, VisCo keeps 91.8% on average, 4 points over VisPruner. Pushed to a single token, the gap widens dramatically: VisCo 85.3%, VisPruner 48.8%, PruMerge+ 35.4%, a lead of 36.5 and 49.9 points respectively.
| Target (LLaVA-1.5-7B) | VisCo | VisPruner | PruMerge+ |
| 32 tokens, avg retained | 91.8% | 87.8% | 77.5% |
| 1 token, avg retained | 85.3% | 48.8% | 35.4% |
On Qwen2-VL-7B, compressing 144 tokens to 18 (an 8x ratio), fine-tuned VisionZip keeps only 81.3% while VisCo holds 90.4%, a 9.1-point lead. Qwen2-VL already has a built-in compression mechanism, so VisCo improving on top of it shows the gain is not just consuming redundancy.
The counterintuitive result: feeding the compressed memory tokens together with the original visual tokens (VisCo+) actually beats the base model, lifting MMB from 67.3 to 69.6 and MMB-CN from 61.8 to 67.8. The authors argue the memory tokens provide a representation complementary to the original visual information, not a simple summary. On efficiency, KV cache drops from 288MB to 18MB (32 tokens) at the same decode speed as VisionZip; in multi-turn QA, because VisCo compresses the image once and reuses it across turns, it overtakes FastV from the third turn.
For anyone deploying VLMs, this is one of the few methods that does not collapse under extreme compression and does not demand a full retrain. Holding 85% at a single token is a strong result, and it opens a path for high-throughput, low-memory settings (many images, long video, on-device). The engineering is appealing: only LoRA and a few dozen memory tokens move, the backbone stays frozen, training is one epoch on 10% of the data, and it is plug-and-play.
The design of letting the VLM compress itself and pouring the layer-wise KV straight into the decoder is reusable. The finding that memory tokens can supply information the original tokens lack (VisCo+ beating the base model) is a useful hint for how to design compressed representations going forward.
The authors concede two things: the encoder adds an LLM-based compression forward pass, which shows up in short-answer settings (MME); future work will add adaptive token allocation and extend to video. So the method is validated on images only, not video.
A concern after reading: the number of memory tokens is fixed (the paper uses 36, 18, and so on), independent of image complexity, so a simple image and a dense image get the same budget. The authors flag adaptive allocation as future work, which is an admission that the current scheme wastes tokens on easy inputs. All results are on mid-scale VLMs (2B/7B) and understanding-leaning benchmarks, with no separate numbers on tasks that demand extreme fine-grained perception (OCR, documents, medical imaging), exactly the tasks most sensitive to token compression. The VisCo+ result (beating the base model) is striking but shown on one backbone and one table, so its robustness across settings is unverified.