RefineAny3D: Depth Refinement as Semantic Alignment for Monocular 3D Detection
Zhihao Zhang, Gengwei Zhang, Tianlong Chen, Xiaoming Liu
cs.CV
2026-08-10
Depth is the monocular-3D bottleneck; the best depth model lowers accuracy. RefineAny3D has a VLM judge projected-box alignment, not predict numbers, lifting three detectors.
Monocular 3D detection (Mono3D) estimates objects' 3D positions from a single RGB image; lacking explicit depth cues, depth becomes the accuracy bottleneck. Open-vocabulary detectors lean on depth foundation models for 3D geometry and generalize well, but the authors find a counterintuitive fact: swapping the strongest current depth foundation model into a strong detector actually drops AP3D by 3.68 on Omni3D, worse than the detector's own prediction, while substituting ground-truth depth raises it by 29.92. So depth is the bottleneck, but training detectors or depth models end-to-end for raw accuracy yields limited returns.
The authors isolate correcting object depth as its own task: given an image and a candidate 3D box from any source (closed-set detector, open-vocabulary detector, auto-labeling tool), output a box with more accurate depth, leaving other attributes alone. The key insight is that depth error has a direct visual signature: when the candidate 3D box is projected back onto the image, a correctly placed box hugs the object, a too-far box projects too small, and a too-close box too large. Depth refinement thus turns from metric regression into an image-space alignment judgment, which is exactly what vision-language models (VLMs) are good at.
In implementation they add six dedicated action tokens to the VLM vocabulary: three directions (depthcloser, depthok, depthfarther) and three magnitudes (stepsmall, stepmedium, steplarge). Magnitudes are scaled by the object's own size (the mean of w, h, l), so one small step means the same thing for a distant truck and a nearby cup. Inference is iterative: the current box is rendered as a wireframe over the image, the VLM does chain-of-thought reasoning and emits a direction-plus-magnitude token pair, the depth is updated, and the loop repeats until the model emits depthok or a step cap is reached. At no point does the model output a numerical depth.
Training data is curated from Omni3D into about 335,000 annotations; each box has its depth perturbed along the camera ray and is paired with a chain-of-thought reasoning trace, yielding about 3 million samples. A two-stage training first freezes the VLM to train only the six token embeddings, then jointly fine-tunes the LLM backbone while freezing the vision encoder to preserve generalization.
This is a clean example of reframing a problem: when an end-to-end model hits a precision ceiling tied to numerical regression, moving the residual problem into a modality the model can reason about visually often works better. For 3D perception practitioners, RefineAny3D is an upstream-agnostic post-processing module that generalizes across categories, scenes, and cameras. It also suggests a broader lesson: rather than forcing a VLM to emit accurate numbers, let it make the discrete judgments it is good at.
Gains concentrate on Easy and Moderate; for small distant objects (Hard) the improvement is limited, because the visual signature of depth misalignment is harder to read there. Iterative inference adds latency, since each step re-renders the wireframe and runs the VLM once, which matters for latency-sensitive settings. Training data comes from Omni3D perturbations, and real noise distributions may differ. The paper gives no direct comparison with other depth-refinement or test-time search methods. The whole approach still depends on a VLM backbone, so end-to-end cost and deployment footprint need evaluation.