JHU's Track, Articulate, Act: Rebuild Articulated Objects from Monocular Video for Robots

Jiaming Zhang and Homanga Bharadhwaj of Johns Hopkins' Brains, Bots, and Behavior lab released Track, Articulate, Act, a framework that reconstructs simulatable articulated objects from a single casually shot monocular RGB video and lets robots replay human manipulation. The work was introduced in a series of posts by @mangahomanga on September 17.

Confirmed

• The pipeline uses SAM 3D for object geometry, SegviGen for part-level mesh segmentation, and 3D point trajectories to recover part motion and fit joints, then aligns hand motion with the object and replays the interaction in MuJoCo
• Demonstrated examples cover everyday home and office scenes—rotating laptop screens, sliding drawers, turning chairs—with joint types and trajectories recovered from estimated 3D point motion; applicable objects also include doors, cabinets, and ovens
• The framework requires no RGB-D, multi-view capture, prior scanning, or robot demonstrations, and reconstructed interactions transfer to an Allegro robot hand in MuJoCo
• The authors note the key difficulty is aligning recovered hand motion with the reconstructed object—small errors in scale or pose hurt transfer quality

Why it matters

• It sharply lowers the data barrier for digitizing articulated objects: a single ordinary monocular video yields a simulatable model, providing a scalable source of objects and interactions for robot manipulation learning in home and office settings

2026-09-17 ~ 2026-09-17 · 5 related posts

• [source] JHU's Track, Articulate, Act reconstructs articulated objects and dexterous interactions from casual monocular videos — mangahomanga · 2026-09-17
• [source] How it works: SAM 3D for geometry, SegviGen for parts, point tracks for joints — mangahomanga · 2026-09-17
• Reconstructed everyday articulated objects: laptop hinges, drawers, chairs recovered from point motion — mangahomanga · 2026-09-17
• [source] Recovering hand-object interactions from 3D point motion, retargeted to Allegro robot hand — mangahomanga · 2026-09-17

1 near-duplicate retellings: mangahomanga