LIFT: Layout-In-Future Video Generation under Large Viewpoint Change via On-Policy Self-Distillation
Shengxiang Ji, Boyang Wang, Haiyang Xu, Bingnan Li, Yucheng Mao, Zeyuan Chen, Xiaojun Shan, Xiang Zhang, Gang Hua, Jianwen Xie, Zezhou Cheng, Zhuowen Tu
cs.CV
2026-09-30
LIFT controls I2V with a camera path and last-frame boxes. Dense-layout OPSD reaches 0.51 mIoU on a 1.3B model vs MagicMotion's 0.41 with dense boxes; 66% user preference.
Camera-conditioned image-to-video models already let you prescribe how the viewpoint moves. Once the camera pulls back or pans, walls, furniture, and streets that were never in the first frame enter the shot. The trajectory does not say what those regions should contain or where objects should sit. Text prompts only give coarse semantics, so large viewpoint changes often invent a pillar or a parked car in the newly revealed space.
Layout conditioning is the obvious complement. Current video methods still ask for per-frame boxes or masks, and they mostly move objects that were already visible at t=0. That is expensive to annotate and does not cover content that appears only after the camera reveals it. LIFT's interface is closer to a storyboard: a first frame, a camera path, and bounding boxes with local text on the last frame that specify the future composition.
The backbone is Wan2.1-Fun-V1.1-1.3B-Control-Camera, trained and evaluated at 352×640 with 81-frame clips at 16 FPS (about 5 seconds). Four conditions go in: the first frame, a global caption, the camera trajectory, and a layout. Camera pose is encoded as Plücker rays (a 6-D ray per pixel), mapped by a light camera encoder into tokens aligned with the video tokens, then added token-wise. Layouts are colored bounding boxes rendered into a pixel-aligned map, encoded by the shared VAE, and concatenated on the channel axis with the noisy video latent and the first-frame latent. Local object prompts, tagged by the same colors, are appended to the caption.
A last-frame-only layout is much harder to learn than dense per-frame boxes. The model has to invent how specified objects enter the shot and how the scene evolves toward the target composition. Vanilla flow-matching SFT on last-frame boxes barely uses the condition. A dense-to-sparse curriculum (D2S-SFT) helps a little and costs a lot of updates. LIFT trains in three stages:
LIFT-Vista is filtered from RealEstate10K, Sekai, and SpatialVID. Clips are kept when FoV expansion, accumulated translation, and DINOv2 patch content-change ratios indicate that the camera uncovers a lot of first-frame-invisible content. Crowded scenes and natural landscapes are dropped. Depth Anything 3 unifies intrinsics and extrinsics. Qwen3-VL-32B boxes the last frame; SAM 3 tracks backward for temporally consistent dense layouts. Stages 2–3 use 58,272 clips; the 600-clip test set is stratified 1:2:2 over three FoV-expansion bins. Mean 5.5 objects per clip, 8,302 categories; 84.8% of clips contain objects absent from the first frame.
Comparisons cover camera control (Uni3C, GEN3C), object motion (MagicMotion), and joint control (Direct-a-Video). MagicMotion and Direct-a-Video receive dense per-frame box trajectories; LIFT sees only the last frame.
| Method | FVD↓ | FID↓ | RotErr↓ | TransErr↓ | mIoU↑ | SRe↑ |
| Direct-a-Video | 539.28 | 71.56 | 28.48 | 2.28 | 0.10 | 0.11 |
| MagicMotion | 277.94 | 18.32 | 15.74 | 1.59 | 0.41 | 0.53 |
| GEN3C (7B) | 89.59 | 15.60 | 3.91 | 2.62 | 0.18 | 0.38 |
| Uni3C (14B) | 111.60 | 12.75 | 3.23 | 0.70 | 0.31 | 0.47 |
| LIFT (1.3B) | 99.35 | 12.84 | 2.97 | 0.59 | 0.51 | 0.59 |
GEN3C posts the best FVD at 89.59; LIFT is at 99.35, ahead of Uni3C at 111.60. FID and LPIPS are nearly tied with Uni3C (12.84 / 0.42 vs 12.75 / 0.42). Camera error and all three layout metrics favor LIFT. Against MagicMotion, which had dense trajectories, mIoU moves from 0.41 to 0.51 and SRe from 0.53 to 0.59.
On the adaptation-stage budget, last-frame SFT and D2S-SFT each use 128K sample updates and reach mIoU 0.44 and 0.47, FVD 114.03 and 110.53. OPSD uses 8K updates and reaches mIoU 0.51, FVD 99.35. Counted from the Stage 1 checkpoint, the SFT routes cost 256K updates, OPSD 136K. Dropping the anchor raises FVD to 129.14. Last-frame-only distillation leaves camera-only FVD at 121.88; camera-only distillation drops last-frame mIoU to 0.38; dual-mode gets 95.66 and 0.51 respectively. Extra layout frames at inference, with no retraining, raise mIoU to 0.62 when all 81 frames are given. A 7-person study (30 trials each, 210 choices) prefers LIFT 65.96%, Uni3C 22.34%, MagicMotion 11.70%.
In virtual production, real-estate walkthroughs, and world-model exploration, people usually care about how the last revealed frame is composed. They will not draw 81 frames of boxes. LIFT compresses a dense-layout teacher into that sparse interface and keeps a camera-only mode on the same weights.
The transferable piece is the recipe: privileged dense conditions as teacher, on-policy velocity matching, prefix high-noise states only, plus a weak SFT anchor. The data point is equally concrete. Large viewpoint change has to be filtered for; on matched 30K-clip sets, filtering cuts RotErr from 15.62 to 13.14 and FVD from 278.85 to 266.58. This is still a 1.3B, 5-second, 352×640 research setup, not a production foundation model.
The paper is explicit that 2D boxes plus local text do not encode depth, orientation, or occlusion, so control stops at roughly inside this rectangle.
Generated-object detection at eval also uses a Qwen model, same family as the annotator, so mIoU and SRe may be optimistic. The user study has seven people. GEN3C still wins FVD by a clear margin; the 1.3B vs 7B/14B quality comparison is not matched on data or resolution, and should not be read as an architecture win. Crowds and natural landscapes were filtered out, so multi-person and wide outdoor scenes are untested. Users pin only the last frame; when objects enter and how they are revealed is left to the model.