Explicit Layer Modeling for Video Object Insertion and Layer Decomposition
Kyujin Han, Seungjoo Shin, Sunghyun Cho
cs.CV
2026-07-28
The TriLayer dataset (3,964 aligned composite/background/foreground triplets) and dual-branch DBL-Diffusion learn explicit RGBA foreground layers with shadows for the first time; users prefer it on both insertion and decomposition.
Video editing has been missing one thing: explicit layered representations. When inserting an object, current methods implicitly generate the foreground inside a mask, which distorts the background and never separates the object's visual effects (shadows, reflections). When decomposing a video into layers, there is no supervision for the foreground layer, so methods fall back on per-scene optimization or implicit inference and generalize poorly. This paper tackles both in one framework with a new supervised dataset.
The core is the TriLayer dataset: 3,964 video triplets, each aligning a composite video, a clean background video, and a foreground RGBA layer (with shadows and reflections). This is the first time layered representations can be learned from supervised composite-background-foreground correspondences. On top of it the authors propose DBL-Diffusion (Dual-Branch Layered Diffusion): two branches, an RGB branch modeling the full scene and an RGBA branch predicting the alpha-channel foreground. The key design is bidirectional cross-attention between the branches in every transformer block, keeping foreground and composite consistent. For training, the RGB branch uses LoRA (rank 128, in-domain) while the RGBA branch uses DoRA (rank 32) for out-of-domain concepts like transparency; each branch samples its own timesteps to balance the two reconstruction dynamics. The base model is Wan2.1-VACE-1.3B. It ships as two models: DBL-Insert (background, text, and boxes in; composite and foreground layer out) and DBL-Decompose (composite in; clean background and foreground out).
On insertion, DBL-Insert beats VACE-Inp, ReVideo, and AnyV2V on background consistency (0.946), subject consistency (0.963), CLIP-I (0.796), and DINO-I (0.747). In a 20-person preference test, DBL-Insert is chosen 61.5% overall versus 17.5% for VACE-Inp. On decomposition, it loses on raw pixel fidelity to the Omnimatte family (those optimize per scene and reach higher PSNR), but it produces plausible backgrounds generatively, and users prefer its foreground quality 75.0% to Gen-Omnimatte's 19.0%.
For anyone in video generation and editing, the triplet dataset is itself a reusable asset; this aligned supervision did not exist before. The method unifies inserting and extracting into one dual-branch diffusion and generates physical effects like shadows explicitly, which means the foreground layer can be tuned separately in post. This is a solid incremental improvement, not a revolution.
Complex physical interactions between objects and scenes, and highly dynamic motion, are still mishandled. The dual-branch design adds compute and memory: the authors report roughly one hour per inference on an A100 80GB, far from real-time editing. On pixel fidelity for decomposition it loses to per-scene optimization, which matters where exact matting is required.