JHU's Track, Articulate, Act Rebuilds Articulated Objects from Monocular Video for Robots

Jiaming Zhang and Homanga Bharadhwaj from Johns Hopkins University's Brains, Bots, and Behavior lab released a new framework, Track, Articulate, Act: using just a casually captured monocular RGB video, it reconstructs simulatable articulated objects and lets robots replicate human operations. The work was introduced in a series of posts by @mangahomanga on September 17.

Confirmed

• The technical pipeline first uses SAM 3D to reconstruct object geometry, SegviGen to segment the mesh by part, then 3D point trajectories to recover part motion, from which joints are fitted; human hand motion is aligned with the object, and the interaction is finally replayed in MuJoCo
• Demonstrated examples cover everyday home and office scenarios: laptop screen rotation, drawer sliding, chair rotation, etc., with joint types and trajectories recovered from estimated 3D point motion; applicable objects also include doors, cabinets, ovens, and more
• The framework requires no RGB-D, multi-view input, prior scans, or robot demonstrations, and reconstructed interactions can be transferred to an Allegro robot hand for replication in MuJoCo
• The authors note the key challenge is aligning the recovered hand motion with the reconstructed object—small errors in scale or pose can degrade transfer quality

Why it matters

• It dramatically lowers the data barrier for digitizing articulated objects: ordinary monocular video yields simulatable models, providing a scalable source of objects and interactions for robot manipulation learning in home and office settings

2026-09-17 ~ 2026-09-17 · 5 related posts

• [source] JHU's Track, Articulate, Act reconstructs articulated objects and dexterous interactions from casual monocular videos — mangahomanga · 2026-09-17
• [source] How it works: SAM 3D for geometry, SegviGen for parts, point tracks for joints — mangahomanga · 2026-09-17
• Reconstructed everyday articulated objects: laptop hinges, drawers, chairs recovered from point motion — mangahomanga · 2026-09-17
• Recovering hand-object interactions from 3D point motion, retargeted to Allegro robot hand — mangahomanga · 2026-09-17

1 near-duplicate retellings: mangahomanga