3DZip cuts 3D vision-language tokens 11× with diversity-anchored selection, keeping 94.7% accuracy

3DZip: Spatial-Aware Feature Diversity-Guided Token Compression for 3D Question Answering

Changwoo Baek, Kyeongbo Kong

ECCV 2026

cs.CV, cs.LG

2026-08-02

3D VLMs choke on massive 3D token sets; 3DZip voxelizes then picks diversity-anchored tokens, holding 94.7% accuracy at 1410-to-128 tokens and 1.9× faster inference.

What problem this solves

3D vision-language models (3D VLMs, which let an LLM understand a 3D scene and answer questions about objects) are bottlenecked by token count: a single point cloud easily produces over ten thousand 3D tokens fed into the LLM, which is slow and blows up memory, and far worse on long videos. Existing 2D token compressors (FastV and the like) port poorly to 3D, because 3D tokens carry spatial structure and a blunt importance cut tends to drop key small objects.

Method

3DZip is a post-hoc, plug-and-play three-stage compressor:

• Coarse voxelization: bin the scene into a δ=0.2m grid and merge tokens within each cell, cutting spatial redundancy first.
• DPP feature-diversity anchor selection: use a Determinantal Point Process (DPP, a sampler that favors subsets both important and mutually dissimilar) to pick representative tokens in each voxel. This is the core novelty: selected tokens cover the feature space broadly instead of clustering.
• Spatially-constrained merging: collapse same-type neighbors within a τg=5 radius.

Against prior spatial-only DPP selection, feature-diversity anchors raise object coverage from 51% to 70%.

Results

At 128 tokens accuracy drops only 2.5 points while inference falls from 342ms to 178ms (1.92×), FLOPs from 9.18T to 0.90T, and KV cache from 722MB to 101MB. It beats FastV, SparseVLM, VisionZip, VisPruner, and DTC, and transfers to Video-3D-LLM and SR-3D.

Why it matters

For 3D VLMs to reach real scenes (robotics, AR), token explosion is an unavoidable engineering wall. 3DZip offers a retraining-free path that cuts memory 7×, meaning the same GPU handles far larger scenes or longer videos. For anyone in embodied or spatial intelligence, this is a directly usable inference optimization.

Limitations

• Small objects suffer most: voxelization and merging bias toward large, dense regions, and sparse small-object tokens get absorbed.
• Hyperparameters (δ, τg, target token count) are fixed, with no per-scene adaptive recipe.
• Validation is mainly on QA (SQA3D); the detail loss from compression is not thoroughly assessed on tasks needing precise localization or segmentation.

Terms

• 3D VLM: a vision-language model that ingests 3D scenes (point clouds, multi-view) and converses, e.g. LLaVA-3D.
• Token compression: trimming redundant visual tokens before they enter the LLM to save compute and memory.
• DPP (Determinantal Point Process): a probabilistic sampler that favors subsets both high-importance and mutually dissimilar, used for diversity selection.
• Voxelization: dividing continuous 3D space into a regular grid of voxels, aggregating points per cell.
• KV cache: stored keys/values during Transformer inference, the dominant memory consumer; fewer tokens shrink it directly.

Source

• arXiv 2608.01185v1
• PDF
• Hugging Face
• GitHub · https://github.com/cvsp-lab/3DZip

Related papers

• VideoGen-Agent learns tool use and lifts VABench from 56.5 to 75.6
• Tencent ARC's 5,000-hour AAA GameHorizon suite finds action planning is the real bottleneck
• WorldCrafter's pose-queryable 3D memory cuts revisit LPIPS from 0.487 to 0.255
• Video diffusion breaks physics because RoPE spatially locks trajectories too early
• Info3R gates CUT3R updates by frame novelty, cutting KITTI ATE 1.68× versus LongStream
• LoG-VGGT keeps VGGT memory at a 4-frame window and mops up pose drift with register tokens

All paper explainers