Xiaohongshu Multimodal Inference Optimization: Vision Token Compression & MoE Re-routing

小红书技术REDtech · wechat · 2026-08-05

Xiaohongshu's tech team detailed their dual-end optimization for multimodal LLM inference in the 'Wen Yi Wen' feature, tackling vision token bloat and MoE decode overhead.

Dynamic Vision Token Compression (VisionZip)

Mechanism: Exploits the fact that deep ViT layers aggregate information into a few dominant tokens. It prunes redundant vision tokens before they enter the LLM based on attention scores.
Global Budget Competition: Vision tokens from all images in a request compete in a unified pool. Information-dense images (e.g., OCR, tables) automatically receive higher token budgets, preventing detail loss from fixed-ratio pruning.
Gains: On a 30-image sample with Qwen3-VL-32B, pruning 40% of vision tokens reduced TTFT by 13% (687ms to 601ms) with negligible performance drop.

Similarity-based Expert Re-routing (SERE)

Mechanism: Addresses the expanding union of activated experts during batched MoE inference. It uses offline expert similarity to dynamically re-route secondary experts to already activated primary experts within the batch, reducing weight fetching overhead.
Key Expert Protection: Introduces a layer-wise 0/1 protection matrix to preserve the original routing of irreplaceable experts, preventing cumulative deviations in structured generation.
Gains: Reduced TPOT by 16.5% in production (8.5ms to 7.1ms). On public Qwen3-30B-A3B, K=2 SERE cut TPOT by 26.1% while retaining 97.7% of original accuracy.

Both methods require no weight updates and are integrated into vLLM with custom Triton/CUDA kernels, fully compatible with PD disaggregation.

Original post →

Xiaohongshu Multimodal Inference Optimization: Vision Token Compression & MoE Re-routing

More from Infra