SmartMage: Dynamic Modality Orchestration for 3D Scene Understanding
Yue Zhang, Yingzhao Jian, Yunqiu Xu, Xiaoxiao Sun, Hehe Fan
cs.CV
2026-08-06
SmartMage selects and routes 3D modalities per query to specialized experts, reaching SOTA on five 3D scene benchmarks and showing that stacking all modalities can hurt.
3D scene understanding (making sense of a room's layout, objects, spatial relations) underpins embodied intelligence. Today's multimodal LLMs (MLLMs) feed in many modalities at once: RGB video, depth, bird's-eye view (BEV), point cloud, voxel. But different questions favor different modalities: "what color is the blanket" mainly needs RGB, while "what shape is the sofa" leans on geometric modalities like point clouds or voxels. Forcing all modalities in equally lets irrelevant ones inject noise and dilute reasoning, and adding more modalities can even lower performance.
SmartMage's take: pick the relevant modalities per question first, then let specialized experts each handle their own.
SmartMage orchestrates modalities with two adaptive layers (global to local), built on Qwen3-VL-8B-Instruct. It first encodes five modalities (RGB, depth, BEV, point cloud, voxel) into a unified embedding space.
The first layer, SMART (semantic-guided modality routing), is a global scheduler. Per instruction it weighs three signals to decide which auxiliary modalities to use: a Semantic Prior Estimator predicts modality preference from the text ("what color" points to RGB, "where" points to geometric); a Semantic Similarity Scorer measures text-to-modality alignment; and a Modality Quality Evaluator scores each modality's reliability from activation strength, sparsity, and stability, down-weighting degraded inputs. The weighted signals form a routing distribution; RGB is always kept as the primary modality and the rest are added on demand.
The second layer, MAGE (modality-aware gating experts), assigns tokens to modality-specialized experts inside the LLM's MoE layers. It predicts each token's modality, then injects that prior into the gating function through a learned modality and expert affinity matrix, pushing tokens toward the right experts. MoE blocks sit at layers 8, 12, 16, 20, 24, and 28, each with 8 experts and top-2 routing.
Training combines semantic alignment losses (pulling the instruction toward relevant modalities and away from irrelevant ones) with modality-attribution and expert-calibration losses and a balancing loss to prevent routing collapse.
SmartMage reaches state of the art on five 3D benchmarks. On 3D QA it scores 32.6 EM@1 on ScanQA and 66.8 on SQA3D, beating the prior SOTA Ross3D by 1.8 and 3.8. On dense captioning it scores 88.7 [email protected] on Scan2Cap, +4.9 over Video-3D LLM. On visual grounding it reaches 59.5 [email protected] on ScanRefer and 60.7 [email protected] on Multi3DRefer, +5.1 and +6.4 over Ross3D.
The ablation backs the adaptive approach. Among fixed modality combos, using all five (row 10) underperforms using fewer (row 8 is higher on several benchmarks), confirming redundancy and interference; SmartMage's adaptive selection wins on every benchmark. Removing the Semantic Similarity Scorer from SMART hurts most (ScanQA 29.8 down to 27.1), marking text-to-modality alignment as the key routing signal.
The authors also build ScanFacet, a diagnostic benchmark that splits questions into eight semantic facets (color, location, material, number, shape, type, spatial relation, other). SmartMage improves on all facets, with +27.1 and +15.9 CIDEr on material and color. Visualizations confirm it learns semantic-modality preferences: color and material favor RGB, spatial and counting favor depth and voxel.
On efficiency it trains in 47.4 seconds per iteration, 2.0 to 2.6x faster than Video-3D LLM and Ross3D, and stays competitive on three RGB-only video benchmarks even without 3D inputs.
Multimodal fusion has long assumed more modalities are better. SmartMage rebuts that with data and offers an interpretable scheme that selects modalities by semantics and splits them across specialized experts while saving compute. For anyone doing embodied perception or 3D and video understanding, it turns "which modality to use" from a hand-set choice into an adaptive one.
The authors note two: token budgets force aggressive keyframe selection (FoVSR) and point-cloud downsampling, which can miss key viewpoints or fine details of small objects; and performance is bounded by training-data quality (blurry images, noisy annotations). A further concern: ScanFacet is the authors' own benchmark, so the semantic-modality preference findings partly depend on its taxonomy and need community follow-up; and on RGB-only video it still trails dedicated video models, so adaptive selection does not replace targeted video training.