Kirin: Animal Motion Generation from In-the-Wild Video
Brian Nlong Zhao, Zhuoyang Pan, James M. Rehg, Jiajun Wu, Shangzhe Wu
ECCV 2026
cs.CV
2026-09-02
Kirin reconstructs ~30k wild quadruped motions into AiM3D and generates from text plus image. Test FID drops from AniMo's 30.0 to 6.25; motions bind onto image-to-3D meshes.
Animal motion lags human motion because capture studios do not scale to most species. Public sets are artist-keyed or ripped from games: few actions, far from the wild. The internet is full of quadruped video, but aligned 3D motion and text are missing. Ponymation trains an unconditional model on horses only. AniMo trains on game animation. Kirin tries to close the loop: reconstruct motion from in-the-wild video, learn a prior, generate from text and image, and bind the result onto an image-to-3D mesh.
Three stages. Reconstruction: on AiM quadruped clips, AniMer initializes a SMAL body (41-D shape, 35 joints) per frame, then a sequence-level pass tightens 2D projection and SO(3) smoothness with cameras held fixed. Global translation comes from SpatialTrackerV2 on the uncropped video, rescaled to SMAL by head-to-tail length. Gemini 2.5 Flash writes six behavior captions per clip. AiM3D has 29,979 motions across 23 quadruped categories and 179,874 captions; 230 clips are test, the rest train. A 100-clip human check marks 86 motions as satisfactory and scores captions 4.62/5.
Generation: MDM plus an image branch. Frozen DistilBERT encodes text, frozen DINOv3 supplies a global image feature, both added to the timestep embedding and fed to a transformer decoder via cross-attention. Text and image are dropped independently at probability 0.2 in training. Inference can be text-only, image-only, or both, with classifier-free guidance. The image is meant to ground species and morphology, not to copy the source clip's motion.
Rigging: Rodin builds a T-pose mesh from one image. SMAL is fitted by Chamfer distance, skinning weights transfer by k-NN, and linear blend skinning applies the generated joint transforms.
On the AiM3D test set, against AniMo, the only public text-driven animal-motion baseline:
| Method | R-Precision Top-3 ↑ | FID ↓ | MM-Dist ↓ |
| Ground-truth | 0.281 | 0.004 | 5.517 |
| AniMo (original data) | 0.089 | 30.043 | 8.564 |
| AniMo (AiM3D) | 0.088 | 30.516 | 8.659 |
| Kirin, text only | 0.097 | 11.889 | 7.007 |
| Kirin, text + image | 0.130 | 6.248 | 6.218 |
Retraining AniMo on AiM3D barely moves FID. The architecture gap does. On AnimalML3D, 1,260 hand-crafted motions neither method trained on, Kirin FID is 138.1 versus AniMo 150.7, better relatively, still far from ground-truth 0.235. Reconstruction: Kirin [email protected] is 0.751 versus AniMer 0.537, and it stays under 1s/frame against SMALify at 30s, but SMALify still leads on [email protected] (0.954) and IoU (0.867 vs 0.698). Qualitatively, AniMo collapses bone lengths across frames; Puppeteer, which inverts a video generator, often produces almost no motion.
This is the largest aligned wild-video, text, and 3D-motion set for quadrupeds, and the generator is the first in this line to condition on both text and image. The animation path can turn one photo plus a sentence into a renderable mesh, useful for previz and game drafts. It is not a biomechanics instrument. FID 6.2 versus 0.004, and R-Precision under half of ground-truth, means draft quality. The real assets are dataset scale and a rig that actually runs.
There is no dedicated limitations section; the numbers draw the border. SMAL is a quadruped template, so birds, fish, and many-legged animals are out. Reconstruction fails on top, front, and back views, occluded legs, very fast motion, and shaky cameras, about 14% of the human sample. Captions are VLM-written: 3 of 100 clips have a wrong action, 7 have small errors in direction or speed. Image conditioning trims Diversity and Multimodality. OOD FID stays in the hundreds. Rigging inherits whatever Rodin and the SMAL fit get wrong. No physics terms, so foot skate and ground penetration are judged by eye. ECCV 2026.