A Plug-and-Play 2D Motion Interface for Real-World Motion Language Models
Kaname Yokoyama, Norimichi Ukita
ECCV 2026 (Oral Presentation
cs.CV
2026-08-17
A plug-in 2D motion encoder from Toyota Technological Institute aligns 2D skeletons to the latent space of frozen VQ-VAE motion-language models with no retraining of the base model; captioning lands within 0.5% of 3D input, and on real monocular video the adapter-equipped 2D path beats 3D estimation pipelines at 17.2 versus 250.2 GFLOPs per frame.
Motion Language Models (MoLMs) tokenize human motion and hand the tokens to a language model for captioning and generation. They are all trained on 3D motion data from mocap or multi-view reconstruction, and their input is a 263-dimensional state vector carrying joint positions, joint rotations, and global displacement. Real deployments can only estimate motion from monocular video, where 3D pose estimation is hard and expensive, and the estimated quality visibly trails the high-quality 3D the models were trained on.
An unverified assumption sits under the whole field: does motion understanding in MoLMs actually depend on the 3D-specific information, depth and joint rotation? If 2D skeletons suffice, deployment gets much cheaper. This paper, an ECCV 2026 oral, runs the verification.
The approach is almost embarrassingly simple: train a 2D motion encoder that maps 2D motion into the continuous latent space of the MoLM's VQ-VAE, with the base model fully frozen and untouched. Training does exactly one thing, minimize an L1 loss between the 2D encoder's output and the 3D encoder's latent features for the same motion. At inference, 2D features pass through the original codebook for quantization and enter the language model as ordinary motion tokens.
Constructing the 2D motion: rebuild the 22-joint skeleton from 3D data, randomize viewpoint (yaw and pitch) during training for augmentation, project orthographically by dropping depth, keep the 13 joints shared by the SMPL and COCO formats, and assemble a 68-dimensional per-frame feature (root velocity, root height, joint positions, orientations, velocities). Training takes a single A100 and 3,000 epochs.
Real video adds a distribution gap: training uses orthographic projection while real observations are perspective with pose-estimation noise. A real-video adapter sits before the 2D encoder and consumes the estimated 2D motion plus per-joint confidence. Its training pairs come from rendering AMASS motions at random viewpoints, running a 2D pose estimator on the renders, and thus obtaining ground-truth 3D plus noisy estimated 2D.
Motion captioning on HumanML3D across three base models:
| Base | 3D input | 2D from scratch | Plug-in 2D encoder |
| TM2T (R-Prec Top-1) | 0.488 | 0.279 | 0.485 |
| MotionGPT (R-Prec Top-1) | 0.516 | omitted, reproducibility | 0.523 |
| MG-MotionLLM (R-Prec Top-1) | 0.583 | 0.487 | 0.576 |
Average gap versus 3D input: -0.2% for TM2T, +0.4% for MotionGPT, -2.8% for MG-MotionLLM, while training from scratch on 2D loses 12.8% to 25.8%. On fine-grained description (FineMotion), the sequence-level gap is -0.5% and snippet-level -1.1%.
Why the gap is so small: token sequences quantized from 2D and 3D agree 44.8% at Top-1 (random baseline 0.2% with a 512-entry codebook) and 67.1% at Top-3. Replacing 50% of 3D tokens with k-nearest codebook neighbors barely moves captioning (-1.4% at k=9) while random replacement collapses it by 34.2%. Neighboring tokens carry similar semantics, so quantization drift does not hurt the language model.
The real-video dataset (132 clips, 10 participants imitating HumanML3D excerpts, monocular iPhone) is where the tables turn. Estimated 3D input (TRACE/WHAM) drops 34-51% against reference 3D, raw 2D input drops 18-47%, and adapter-equipped 2D drops 10-22%, even gaining +4.3% on TM2T. Compute is lopsided: WHAM costs 250.2 GFLOPs per frame versus 17.2 for ViTPose-Base plus the adapter, a 14.5x gap.
This is a template for any modality model trained on 3D but deployed on 2D: no base-model retraining, no distillation, one bolt-on alignment encoder swaps the input representation. The methodological lesson is that latent-space alignment vastly undercuts end-to-end retraining, which reruns the whole pipeline and still loses ten-plus points, while an L1-aligned encoder compresses the gap to single digits.
There is a deeper reading. If 2D skeletons approximate the MoLM motion representation this well, motion understanding in these models rests largely on view-invariant 2D-observable structure, and 3D rotation and depth contribute little to understanding what a motion is doing. For action recognition, human-computer interaction, and video understanding applications, the cheap and mature monocular 2D pose front end can now feed state-of-the-art MoLMs directly.
The authors' own list: the adapter's correction smooths motion sequences and suppresses fine-grained differences, hitting newer MoLMs that generate detailed captions harder (still -18 to -19% on real video); failure cases include a lost S-shaped walking trajectory and hands raised to the face described as clapping, an over-generalization; the dataset relies on human imitation, so imitation error injects semantic mismatch.
Further concerns from a close read. The paper scopes its own conclusion carefully: 2D sufficiency is observed under a setting where 2D features are explicitly aligned to the 3D latent space, and does not establish that 3D-specific information is unnecessary in general. Motion generation is untouched; only motion-to-text was tested, and the reverse direction inherently needs 3D output. The real-video dataset is small at 132 clips, and its 86.4% video-caption semantic consistency means the benchmark itself carries over a tenth of label noise. The alignment encoder still needs paired 3D-2D data derived from 3D projections, so purely annotation-free settings are not covered.