Beyond Selection: Token Parameterization for Extreme Visual Token Compression
Rui Zhong, Yu Li, Zheyu Yan, Cheng Zhuo
NeurIPS 2026 (Spotlight)
cs.CV, cs.LG
2026-09-28
Braco, a four-step DCT-plus-residual visual-token coder, keeps 95.2% of uncompressed LLaVA accuracy at 23× compression, with 16.6× lower compressor latency than QueCC.
VLMs turn an image into hundreds or thousands of visual tokens, and LLM prefill cost grows linearly with that length. Mobile inference, low-latency agents, and long-context multimodal chats often have room for only a few dozen visual slots. Token pruning and merging still pay off at moderate ratios. At extreme budgets they drop rare but task-critical patches, and raw attention is a shaky importance score.
Learned resamplers take a different route. Perceiver-style cross-attention or learned queries distill the dense grid into a fixed latent set. Accuracy is usually higher. The bill lands inside the compressor: extra attention, extra parameters, and staged multimodal alignment. This NeurIPS 2026 Spotlight from Zhejiang University recasts extreme compression as token parameterization. Which subspace you keep (compressibility) is a separate choice from how you arrange coordinates inside that subspace (learnability).
Braco stands for Backbone-residual plus basis plus coordinate. A vision encoder emits an N by N patch grid, 24 by 24 equals 576 for LLaVA-1.5. A four-step coder g maps that grid to K tokens:
The residual stays spatial because a localized spike spreads across DCT coefficients. Even the best m frequency bins recover little of that energy when the spike is sparse.
On LLaVA-1.5-7B, Acc. is the mean of eight benchmarks normalized by the 576-token Vanilla interface (8.67T FLOPs, 67.25ms prefill). Baselines cover pruning and merging (PruMerge, DivPrune), learned interfaces (MQT-LLaVA, QueCC, TokenPacker), and transform coding (Fourier-VLM).
| Method | tokens | ratio | Acc. | FLOPs | latency |
| Vanilla | 576 | 1× | 100.0 | 8.67T | 67.25ms |
| Braco | 25 | 23× | 95.2 | 1.37T | 41.03ms |
| TokenPacker | 25 | 23× | 94.2 | 1.38T | 38.40ms |
| Braco | 16 | 36× | 94.0 | 1.25T | 40.59ms |
| QueCC | 16 | 36× | 93.9 | 1.36T | 63.22ms |
| Braco | 9 | 64× | 93.2 | 1.15T | 40.22ms |
| QueCC | 9 | 64× | 93.1 | 1.26T | 62.79ms |
| Braco | 4 | 144× | 91.2 | 1.09T | 40.97ms |
| QueCC | 4 | 144× | 91.4 | 1.20T | 64.17ms |
From 23× to 64×, Braco sits on the accuracy-cost frontier. At 144× it trails QueCC by 0.2 Acc. and still runs faster. Prefill FLOPs fall from 8.67T to 1.15T-1.37T, an 84.2%-86.7% cut versus Vanilla. Isolated at 16 tokens, the compressor itself is 1.073ms and 1.389G against QueCC at 17.864ms and 109.504G: 16.6× lower latency, 78.8× fewer FLOPs.
Per benchmark at 25 tokens, MMBench-EN is 65.4 versus Vanilla 65.5. TextVQA falls from 58.0 to 52.3, so reading text in the image takes a clearer hit. PruMerge on the same budget is at 80.2 Acc.; pruning has already collapsed. The closest transform cousin, Fourier-VLM, scores 92.9.
Larger inputs widen the saving. Vicuna-7B at 2880 visual tokens, still 36× down to 80, gives Braco Acc. 98.1, with the pipeline dropping from 40.57T to 3.45T and from 261.08ms to 44.36ms. Qwen2.5-3B at 3645 tokens down to 20 (182×): Braco holds 90.8 Acc., QueCC drops to 68.9. Query-conditioned compression does not amortize well on the smaller LLM; at 729 and 1024 inputs QueCC is slower than the uncompressed baseline.
Ablations match the factorization. At 9 tokens the hybrid c2s5 scores 93.2 against 89.8 for backbone-only c3s0 and 92.0 for residual-only c0s9. Adding five residuals to a C=2 backbone lifts Acc. from 85.7 to 93.2. At 16 tokens, swapping DCT for spatial or Haar costs 2.3-3.0 points; the wrong coordinates or a random rotation cost 2.5-3.1; replacing Polar embeddings with learned or 2D sine-cosine codes costs 0.8-0.9. Across three matched seeds, Braco leads QueCC by 0.24 and 0.27 Acc. at 16 and 9 tokens, with standard deviations at most 0.14.
This is a careful incremental step: classical transform coding plus a cheap spatial residual, wired as a VLM visual interface. The payoff is specific. Cutting visual tokens from hundreds to tens is what actually shrinks LLM prefill. The interface ignores the prompt, and the compressor stays in the GFLOP range, about an order of magnitude cheaper than query resamplers such as QueCC. On 8x A100, 16-token two-stage training takes 0.4h pretrain plus 6.5h instruction tuning, shorter than Vanilla (3.5 plus 10h) and no slower than QueCC. Code is public.
On-device VLMs, long-context multi-image inputs, and AnyRes-style high-resolution slicing are the settings where this is worth trying. The paper does not claim a drop-in for video, dense localization, or non-LLaVA recipes.
Evaluation is mostly LLaVA-family encoder-to-LLM pipelines and standard single-image benchmarks, so that every learned baseline can be retrained under one open recipe. The budget-dependent coordinate rule is calibrated on the tested encoders and resolutions. New backbones, video, dense localization, domain shift, irregular region features, or dynamic visual tokens may need a small sweep over the backbone-residual split and coordinate layout. Three-seed comparisons cover only the key Braco-QueCC and Braco-TokenPacker budgets; remaining Acc. numbers are one checkpoint each.
A few more caveats sit in the tables. Acc. is a Vanilla-normalized mean, not a raw score. At 576 tokens, full-pipeline latency still includes vision encoding and prompt processing that barely move with token count, so end-to-end gains show up more clearly on high-resolution inputs. CelebA linear probes only show that frozen coordinates remain readable; they do not replace the VLM table. QueCC is omitted at 25 tokens because its native grid cannot emit 5 by 5 on a fixed 24 by 24 lattice, so that cell has no direct comparison.