Geometric and Semantic Coupling for Interaction Understanding in 3D Scenes
Hanyang Kong, Xingyi Yang
cs.CV, cs.AI
2026-09-22
Segment-Snap picks hinge sides from predicted handles, with no motion head. Validation motion AP rises from 13.74 to 40.98 and leads the public Articulate3D challenge.
Interacting with a 3D scene takes more than naming the cabinet. The system has to mark which surface moves, how it moves, and where a hand should grab. Doors are large; handles may be a handful of points. A closed panel is compatible with a hinge on either side. Better part masks do not automatically yield better motion, and accurate handle locations do not by themselves decide rotation versus translation.
Articulate3D annotates those three outputs on ScanNet++ reconstructions. Methods such as USDNet predict parts, interactable regions, and motion jointly, yet the motion head does not use detected handles to pick a hinge. Segment-Snap treats the physical link as evidence that can move once, in one direction, at inference.
Three independently trained Volt-B voxel Transformers read the same RGB point cloud with normals. A part predictor emits movable surfaces and a rotation/translation class, with no axis regression. A dense handle predictor classifies points and groups them into instances; those locations later guide motion. A joint predictor decodes child handle masks from its own part queries; only those extra handles enter the final set.
Motion decoding is a training-free geometric rule. Each part mask keeps its largest connected component and is fit with a minimum-area planar rectangle. Rotational axes use a world-vertical prior; translation axes use the box's thinnest direction. The nearest handle centroid within 0.5 m selects, among four axis-parallel candidates, the hinge line farthest from the handle. The origin is the support centroid projected onto that line. Missing handles fall back to the centroid. Horizontal hinges and tilted mechanisms are blind spots of the prior, not cases the handle cue is meant to fix.
The reverse transfer relabels only the extra child handles. A confident containing part with at least 0.9 overlap can overwrite the class; otherwise a dense handle is the fallback. Masks and scores stay put, and dense detections are never edited. Final handles are not fed back into motion decoding. There is no iteration.
On 42 public validation scenes, holding masks, classes, scores, and axes fixed, swapping the centroid origin for a handle-guided hinge raises motion-gated AP from 13.74 to 40.98 (+27.25). The entire gain sits on rotations, 1.88 to 56.37; translation origins are not scored. Appending joint-model children lifts handle AP from 24.63 to 29.65; part-only relabeling adds 0.98, and the full rule reaches 30.99. Part AP in the reference config is 47.93.
Against published USDNet numbers, parts look higher and handles are close (30.99 vs 31.1), but scene lists, training resources, and evaluator details are not matched.
On frozen part features the training-free rule scores 40.98; the closest learned selection head reaches at most 39.36 across three seeds, with a paired interval that includes zero, so the paper does not claim the rule beats learned heads. Spatially permuted or random children add nothing; real children recover 53 extra ground-truth handles beyond 140 already matched by dense detections.
On the public test set, team TnG reports 48.28 motion AP and 34.46 handle AP, first among public entries (runners-up 40.56 and 32.86). Test labels are withheld. All mechanism studies use validation, and the part checkpoint was picked on validation motion AP.
A dedicated motion-regression head is not the only way to recover interaction structure. Where the handle sits is often enough to pick the hinge side of an upright door. For pipelines that feed simulation or manipulation, that is cheap geometric structure: axes and origins come from fitting and rules, and learning concentrates on finding parts and handles. Handle-AP gains are smaller and less stable. Motion is the main result.
The decoder assumes roughly planar parts and upright rotational axes. Missing handles, fragmented support, and a neighboring handle can place the hinge on the wrong side. Class correction varies across child-training seeds from 0.19 to 1.34 points, so the reference +1.34 is not a stable share. The child-union gain shows a complementary proposal source; it does not isolate parent-query conditioning. Validation is used for both development and ablation, so it is not an untouched generalization estimate. On the 12 non-vertical hinges in the qualitative set, the handle cue cannot repair the axis prior.