BLARM: Animating 3D Objects from Video via Blending Latent Rigid Motion Primitives
Pradyumn Goyal, Yizhak Ben-Shabat, Hsueh-Ti Derek Liu, Haomiao Jiang, Snehasish Mukherjee, Kyle Spence, Mark Stauber, Evangelos Kalogerakis, Yunze Zeng
cs.CV
2026-09-01
BLARM animates a mesh from video with 31 latent rigid primitives, cutting ActionBench CD-3D from 3.30 to 1.71 and 16-frame runtime to 3.13s.
Static 3D generators already emit usable meshes. Animation still sits on two weak tracks. Geometry-only riggers infer a skeleton or cage that often fails to match the motion in a particular video. Feed-forward vertex regressors skip the rig and predict a displacement for every vertex, which is high-dimensional, temporally noisy, and awkward to drop into a production animation stack.
Monocular video is the cheap motion source. The missing piece is a deformation space that follows the video, keeps vertex correspondence, and does not ask for skeletons, cages, or skinning labels.
BLARM factors animation into two parts. Time-varying: J latent rigid components, each decoded per frame into an SE(3) transform. Component 1 is a global root; the rest move in the root frame. Time-invariant: per-vertex skinning weights over the non-root components, softmax-normalized and non-negative. Linear blend skinning mixes the local transforms, then the root transform is applied.
The representation is more adaptive than a fixed skeleton and more compact than per-vertex offsets. J=31 non-root components is the operating point.
A frozen TripoSG encoder turns the canonical mesh into dense geometric features. Learnable queries compress them into J deformation latents. Those latents are broadcast across time and refined by factorized blocks: cross-attention to that frame's DINOv3 tokens, spatial self-attention among components inside a frame, and temporal self-attention along each component. Cost is O(TJ²)+O(JT²) rather than full spatiotemporal attention. Skinning weights come from a vertex descriptor (coordinates, normals, frozen PartField features) attending to temporally pooled motion latents. No bone or weight supervision.
Training mixes three losses: a motion-error-weighted L1 reconstruction term, entropy on the weights so each vertex binds to few components, and a trajectory-based contrastive term so similarly moving vertices share similar weights. Loss weights are 3 / 0.001 / 0.3. Data is about 10k Objaverse animation sequences with evaluation assets held out. Main tables use ground-truth canonical meshes; the in-the-wild split reconstructs the first frame with Trellis2.
ActionBench has 128 clips of 16 frames. Motion80 has 80 variable-length clips. Baselines take the same video-plus-mesh input and do not assume a predefined rig.
| Method | ActionBench CD-3D | CD-Motion | FVD |
| Mesh4D | 2.57 | 9.24 | 787.38 |
| Motion3-to-4 | 2.49 | 10.31 | 1270.80 |
| ActionMesh | 3.30 | 11.12 | 1126.90 |
| BLARM | 1.71 | 7.15 | 426.72 |
On Motion80, CD-3D falls from 2.25 (Motion3-to-4) to 1.93 and FVD from 735.57 to 482.50. On Consistent4D, with reconstructed meshes, FVD is 890.96 versus 1410.61 for ActionMesh; CLIP ties Mesh4D at 0.84. Sixteen frames take 3.13s on an RTX 4080, against 350s for ActionMesh and 55.33s for Mesh4D.
Ablations pin the contrastive term. Dropping it collapses the model to about one active non-root latent per shape and raises CD-Motion from 7.15 to 12.90. Direct per-vertex offsets score 10.96 on the same metric. J=15 underfits; J=127 fragments and CD-Motion rises to 11.88.
For 4D asset and game pipelines, this is a feed-forward path to editable mesh animation without authoring a rig. Predicted weights often land on coherent moving regions such as a bending shoe or a group of feathers, so the model is discovering a part-like basis on its own. The runtime is in interactive range.
The gain is still incremental. A canonical mesh is required, motion is a blend of rigid parts, and strong non-rigid effects such as cloth are out of scope.
The authors flag two failure modes. Nearby regions that look similar but should move independently can share a component and lock together. If the input mesh topology cannot support the target motion, the deformation produces artifacts.
A few gaps stay open. Training and the main tables use synthetic Objaverse renders; the wild split reports appearance only, so geometric error is unknown. Ground-truth meshes in the main protocol hide reconstruction noise that will show up in a real capture-to-animate loop. Dual quaternion skinning is mentioned as a drop-in for LBS and never compared. The contrastive loss needs ground-truth trajectories, so it does not transfer to video-only supervision.