Three-step hand video keeps MANO Dice at 92.49 and hits 18 FPS on one H200

LynnReal-Hand: Unlocking World Interaction with Hand Control

Shaohao Rui, Chaoyang Zhang, Xiaofeng Mao, Weijie Ma

cs.CV, cs.AI

2026-11-17

Trained on 100K clips, LynnReal-Hand hits FVD 181.67 and MANO Dice 93.02; its three-step model keeps Dice at 92.49, streaming 540p at 18.03 FPS and 0.741s latency on one H200.

What problem this solves

Interactive video world models already follow text and a moving camera. Genie, Matrix-Game 3.0, and PAN made "where to look" and "where to go" work. Grabbing an object asks for something else. Reaching, closing the hand, letting go, and using both hands are specified on fingertips and contact patches. A navigation command does not name them.

HandWorld, Hand2World, Generated Reality, and JointControlVideo already condition video generators on articulated hands. Three failures still stack. A hand covers little of the frame, so a global loss barely pays when a finger is wrong. Squeeze the model into a few-step causal generator and each new chunk conditions on the model's own previous output, so local motion errors grow. After denoising is fast, decoding a high-resolution latent is still slow.

Method

Training uses 100K clips, real and synthetic, indoors, outdoors, and sci-fi. A vision-language model writes captions, VIPE estimates the camera, and WiLoR fits MANO on every frame. MANO is a parametric hand: pose and shape build a 3D mesh, rasterized as a conditioning video, red for the right hand and blue for the left. HandBench has 200 sequences. The 160 in-domain clips come from eight training sources. H2O and TACO are held out entirely and make a 40-clip out-of-distribution split.

The generator is a bidirectional Diffusion Transformer in the frozen latent space of the H3 VAE. The first frame and the text go through Qwen3-VL, hand motion is tokenized by the H3 VAE, and camera pose is injected with PRoPE, which writes projective geometry into positional codes. Conditioning encoders stay frozen. Only the DiT is trained. The base objective is conditional flow matching, a velocity field between noise and the clean latent. One weight covers the whole frame, so errors on the few hand tokens are easy for the background to swamp.

Interaction-Preserving Training has three stages and two failure modes in view: a weak hand signal in pretraining, and lost hand tracking after acceleration.

Ultra-Fast-VAE keeps all 26 blocks and the 2048-dimensional residual stream, cuts attention heads from 32 to 8 and SwiGLU width from 8192 to 1536, and ends at 361 million parameters. Channels are scored by activation strength times the downstream projection norm. The narrowed output projections are initialized by ridge regression, then the compact attention and feed-forward branches are distilled against the full-width operators on the student's own hidden states. Decoding keeps no KV cache across chunks. A temporal loss upweights the hand, with a separate term on boundary frames, because fast fingers flicker where chunks meet. Each chunk reads five current latent planes, does not look ahead, emits 17 RGB frames, and is exported at W8A8.

Results

On the eight in-domain sources, Base and the baselines generate at 768p. Fast, the streaming variant, runs at 540p. All scores are computed against 768p reference frames.

MethodFVD↓MANO Dice↑Cam-BG↑PSNR↑
JointControlVideo, not retrained358.3489.9323.2116.68
Generated Reality, fine-tuned348.2689.1126.6517.03
Hand2World bidirectional, fine-tuned294.3290.6054.1319.64
Hand2World streaming, not retrained963.9676.2317.1712.92
LynnReal-Hand-Base181.6793.0264.9720.90
LynnReal-Hand-Fast, 3 steps237.3392.4949.3519.12

MANO Dice is the overlap between a hand silhouette refit on the generated frames and the ground-truth mask. Cam-BG is agreement between background motion and the reference camera. Against Hand2World bidirectional fine-tuned on the same 100K clips, Base moves FVD from 294.32 to 181.67, a 38.3% relative drop, Dice from 90.60 to 93.02, and Cam-BG from 54.13 to 64.97. Fast reaches FVD 237.33, still under every offline bidirectional baseline, with VBench subject consistency 95.27 and background consistency 95.34. Hand Dice sits 0.53 below Base. Camera following drops further, to Cam-BG 49.35.

On held-out H2O and TACO, Base records FVD 434.46 and 398.03, against 879.73 and 762.02 for fine-tuned Hand2World. The paper puts that relative cut above 47%. Dice is 93.70 and 92.84. Fast is not retrained. On H2O it scores Dice 92.78 and FVD 690.19. Out-of-distribution FVD is still more than double the in-domain figure.

On one H200, 33B Fast plus the slim decoder streams 540p in 17-frame chunks at 0.741 s latency, 18.03 FPS, and 34.89 GiB peak memory. The same 33B Base model, run over a full sequence, takes about 770 s at 0.136 FPS and 82.59 GiB. Streaming Hand2World, at 5B, reaches 8.65 FPS and 33.44 GiB, with in-domain Dice of only 76.23.

Decoder-only timing uses 17 frames at 1344×768. Ultra-Fast-VAE has 361.2M parameters, 34.43 dB PSNR, 46.77 ms latency, and about 363 FPS. Native H3 has 2423.5M parameters, 35.04 dB, and 831.30 ms: about 6.7× fewer parameters, 17.8× faster decoding, and 0.61 dB lower PSNR. Against LightVAE, PSNR is 0.70 dB lower and latency about 12.5× lower. Against Tiny-H3, PSNR is 4.03 dB higher and frame-difference error falls from 13.78 to 9.80.

Blind pairwise judgments cover motion and visual realism together. Base wins 70.5% against fine-tuned bidirectional Hand2World. Fast wins 58.5%. Both win 100% against streaming Hand2World. Base wins 93.5% and 89% against JointControlVideo and Generated Reality.

The ablations locate where the loss has to look. Stage I uses 100 HandBench clips drawn apart from the main split. Global flow matching alone scores Dice 87.02 and FVD 473.69. A hand latent term adds 2.67 Dice. LPIPS then drops FVD from 442.33 to 358.62 and lifts Dice to 92.43. DINOv2 finishes at 351.55 and 92.74. Stage III uses 50 clips with a hand in every frame, all under a three-step budget. No distillation: FVD 404.68, Dice 89.60. High-noise video scoring (shift = 12) reaches Dice 92.03. Letting the image branch touch the hand reaches FVD 208.75 while Dice stalls at 92.00. Restricting that branch to the background reaches Dice 92.50 and FVD 198.99. An unmasked image teacher improves the global look and does not improve hand tracking.

Why it matters

Fast is the run that can actually be used: three denoising jumps, a two-chunk window, and a cache-free slim decoder, with near real-time throughput at 540p on one GPU. The paper lists code and model pages, and says the datasets, weights, and full training and inference code will be released. The practical barrier is the 33B backbone.

Against Hand2World fine-tuned on the same data, the offline gain is incremental: a little over two Dice points, and a 38.3% relative drop in FVD. Streaming is where the gap opens. The other streaming checkpoint falls to Dice 76. Fast stays at 92.49.

The hand trajectory and the camera are supplied from outside. The model returns a coherent video. Contact does not rewrite object state, and the frames do not go back to a controller. It does not yet work as a simulator for dexterous manipulation.

Limitations

The two-chunk causal window has no long-term 3D memory. When an object leaves the frame and comes back, nothing in the model is charged with keeping it. Kinematic control is not coupled to a physical contact prior. Closed-loop robot learning is left for later.

Dice measures silhouette overlap. It does not say whether the grasp held, or whether the object moved in a way physics would accept. Fast is generated at 540p and most baselines at 768p, then everything is scored on 768p references, so pixel metrics are not on equal footing. JointControlVideo and streaming Hand2World were not retrained on the 100K clips. The 0.741 s figure is the wait for a 17-frame chunk. 18 FPS is throughput. For hand interaction that wants feedback within a few hundred milliseconds, that lag is still large. In-domain FVD is 181.67. On H2O it is 434.46. The 100-clip and 50-clip ablations are not the main split. Error from WiLoR and VIPE is not measured on its own, and the preference study does not report how many raters or items were used.

Terms

Source

Related papers

All paper explainers