Messages, Not Tokens: Grounded Coresets for Faithful VLM Compression
Long Qian, Jiaqi Wei, Bingke Zhu, Yingying Chen, Jinqiao Wang
cs.CV
2026-08-03
Training-free GMC cuts 80% of Qwen2.5-VL-7B's visual tokens yet keeps 97.78%–100.36% of full capability, beating Top-K baselines by preserving signed attention messages.
Vision-language models slice each high-resolution image into a long strip of visual tokens, and every token runs through the language decoder and sits in the prompt KV cache. A single document page can spawn tens of thousands of them, so inference gets slow and expensive.
Nearly every existing compressor (FastV, VisionZip, DivPrune, MMTok) does the same thing: score each token independently and keep the Top-K. That misses something. A text query does not read isolated patches; it consumes a signed attention message, a weighted sum over the visual population in which equal-magnitude contributions pointing opposite ways cancel. Independent Top-K piles onto one high-response region, drops sparse but complementary evidence (a bar plus its axis label, say), and discards whatever the removed tokens were carrying.
GMC (Grounded Message Coreset Pruning) is training-free and inserts one step before a chosen decoder layer. It splits compression into two coupled questions: where to keep carriers (support allocation), and what each carrier should carry (message realization).
For support allocation, instead of one scalar per token, GMC treats the carriers a query actually needs as three client families to cover. Query-grounded clients capture the visual content the text query is already attending to. Appearance clients capture structure not yet activated but possibly needed later, via normalized feature prototypes with Gibbs similarity. Spatial clients preserve coordinate geometry through radial landmark kernels. Coverage is a weighted facility-location objective that the authors prove is normalized, monotone, and submodular, so a batched greedy solver has provable guarantees; once a region is represented, its redundant high-score neighbors lose all marginal value, which kills duplicate stacking at the source.
For message realization, each deleted token is assigned to a chosen carrier by semantic, local, and spatial similarity. The carrier becomes a weighted centroid that preserves the signed first moment of the population it absorbs, so cancelling opposite-direction contributions are kept rather than collapsed to magnitudes. RMS restores scale. Carriers stay at their original multimodal positions, and the decoder's native compact attention recomputes the signed message downstream, nothing is re-derived after the fact.
Two operating points: GMC-H2 compacts before block 2 (token-layer work drops to about 16%, compute-cheap), GMC-L16 before block 16 (work about 61%, higher fidelity).
Main backbone Qwen2.5-VL-7B (N=1296 visual tokens), Full-relative scores (full = 100):
| Method (K=256, down 80.2%) | Avg. |
| FastV | 92.86 |
| DivPrune | 92.44 |
| VisionZip | 92.16 |
| MMTok | 93.40 |
| HAWK | 94.61 |
| GMC-H2 | 97.78 |
| GMC-L16 | 100.36 |
At the tighter K=128 (down 90.1%), FastV scores 82.98, MMTok 83.86, HAWK 87.53, while GMC-H2 reaches 93.51 and GMC-L16 99.11. The gap widens as the budget shrinks.
On hallucination GMC-L16 is cleaner than the full model: POPE 87.04 vs full 87.07, AMBER 87.17 vs 86.33, CHAIRs 28.80 vs 37.60, CHAIRi 7.41 vs 9.05, all beating VisionZip and MMTok. Transfer to LLaVA-1.5-7B without retuning hits 99.76 and 99.86 at K=64 and K=128. On a 15,876-token DocVQA page, GMC-H1 keeps 98.87% ANLS with 1.258x end-to-end speedup and 73.94% prompt-KV reduction.
Controlled ablations isolate support construction as the dominant factor and candidate-logit error as the best predictor of harmful flips. With all post-selection operations fixed, GMC still beats four selectors by 2.74 to 12.61 points at L16.
VLM inference is expensive mostly because visual tokens are numerous. A training-free method that drops 80% of them without losing accuracy means existing Qwen and LLaVA checkpoints can shed KV cache and decode cost by inserting one layer, with no retraining.
The conceptual payoff matters as much as the speed: compression should preserve the signed message a query consumes, not high-scoring tokens. That explains why independent Top-K collapses at small K, it has been preserving magnitudes while ignoring signs. Anyone deploying VLMs on long documents or high-resolution imagery gets a usable lever.
Fewer tokens do not mean proportionally less wall-clock time. Token-layer work drops sharply, but the client-construction, selection, transport, and compaction pipeline has its own cost, so the measured DocVQA speedup is only 1.176 to 1.258x, far less dramatic than the 80 to 90% token reduction.
GMC-H2 at K=128 falls to 93.51%, so the most aggressive compute savings cost quality; L16 keeps fidelity but only trims token-layer work to about 61%. Scores above 100 are relative to a shared full-model baseline for that backbone, not superhuman performance. MME carries the widest confidence interval due to its resampling scheme, and the spatial bank acts as a low-weight prior with small marginal effect in some configurations.