OmniPack: Unified Token Compression for Efficient Omni-modal Large Language Models
Wanshun Su, Yang Shi, Feihu Liu, Ziwen Yu, Yan Min, Zhuoran Zhang, Qixun Wang, Haotian Wang, Shixuan Liu, Yuanxing Zhang, Peng Wu, Chengfu Huo, Liang Ding
cs.CV
2026-08-04
OmniPack compresses audio-visual tokens for omni-LLMs training-free, in two stages before and inside the LLM; on Qwen2.5-Omni-7B it keeps 15% of tokens at 95.6% performance and 10% FLOPs.
Omni-LLMs (Qwen2.5-Omni, GPT-4o, Gemini 3.1 Pro: models that jointly handle vision, audio, and text) are powerful, but dense audio-visual tokens impose heavy compute and memory cost. Existing token compression degrades badly at low budgets: compressing before the LLM discards structurally important and globally distributed evidence, while compressing inside the LLM underuses query-conditioned audio-visual collaboration. OmniPack targets both gaps.
OmniPack is training-free and works in two stages.
Hyperparameters: layer 18 for the inner stage on 7B and MiniCPM-o, layer 26 on 3B; a sensitivity analysis is provided and the settings are robust.
On Qwen2.5-Omni-7B across five benchmarks (AVUT, WorldSense, DailyOmni, VideoMME, LVOmniBench):
| Token retention | Performance | FLOPs |
| 25% (pre-LLM only) | 98.2% | 4.9x reduction |
| 15%/7.5% | 95.6% | 10.0% of original; 4.5x prefill speedup |
| 10%/5% | 92.9% | 6.8% of original |
It beats SEATS (both variants), OmniSIFT, OmniZip, VisionZip-om, and FastV-om across retention settings. It holds across backbones: Qwen2.5-Omni-3B keeps 92.7% at 15%/7.5% (9.0% FLOPs), and MiniCPM-o-2.6 keeps 100.8% (10.0% FLOPs), slightly above the baseline. Ablations show coverage selection is the strongest single component on WorldSense and LVOmniBench, the three components are complementary, audio-visual collaboration beats independent per-modality compression, and layer 18 is optimal.
For teams deploying omni-LLMs, this is a training-free, plug-and-play way to cut compute; a 10x FLOPs reduction matters most for long audio-visual inputs. The two ideas of merging rather than discarding, and splitting redundancy handling into a pre-LLM and an inner stage, transfer to other multimodal architectures. The MiniCPM-o result above 100% suggests compression doubles as a denoising regularizer, an extra payoff.
The paper has no dedicated limitations section. A few concerns: the FLOPs metric counts only the vision-plus-audio token contribution inside the transformer, excluding encoders, projectors, text, and the output head, so end-to-end speedup is less dramatic; the MiniCPM-o gain above 100% means compression acts as regularization, but whether that benefit is stable across all tasks is unclear; hyperparameters are tuned, though a sensitivity analysis is given; the benchmarks emphasize audio-visual understanding and do not cover generation tasks.