HKU's JoHan recovers smooth 3D hand motion from video via generative 2D-3D joint modeling

HKUCDS · hf · 2026-10-13

Researchers from HKU and UC Berkeley propose JoHan, a unified generative framework that recovers 3D hand motion directly from video without intermediate per-frame pose predictions. Trained from scratch, it jointly generates aligned 2D and 3D local hand pose sequences: the generated 2D trajectories leverage spatial-temporal image cues to guide 3D reconstruction, while a learned motion prior enforces temporal consistency; the 2D-3D correspondence also recovers global hand position and orientation relative to the camera. Experiments show markedly better accuracy and speed on local hand-pose and camera-space reconstruction, with much smoother motion dynamics than prior methods.

Original post →

More from Embodied

Embodied channel →