PhysInsert: Physically Plausible Object Insertion with Geometric Priors and Bidirectional Object-Scene Interactions
Yue Gong, Le Zhang, Rui Chen, Song Chang, Xiaochao Qu, Luoqi Liu, Ting Liu, Yebin Liu
cs.CV
2026-10-30
PhysInsert adds a VGGT geometry stream to Qwen-Image-Edit, trained on 12,951 counterfactual pairs. Text-guided DIoU rises from 48.32% to 62.99%.
Reference-based insertion has to do more than paste an object in. Perspective, pose, and illumination need to match the target scene, and the shadows and reflections the object causes need to show up, while its identity stays intact.
Supervision is thin in three places. Cross-lighting data rarely show the same object under a controlled light change, so models copy the reference look. Mask-and-fill wipes the edit region and repaints it. Object shading can shift, but shadows, mirror images, and reflections in the scene have no aligned target. Pairs built by generative editing or cut-and-paste do not guarantee those effects were rendered. Simple backgrounds also skip occlusion and contact. Across viewpoints, fixed-camera pairs barely teach perspective or pose, and a mask or box leaves little room for text to set placement. A 2D image loss never sees how the object sits in the scene's 3D layout.
InsertWorld starts from complete BlenderKit scenes and deletes an object that is already there. Camera, lighting, and everything else stay fixed. Only object presence changes, and the renderer recomputes shadows, mirror images, and reflections. Random drops tend to float or interpenetrate. Deletion keeps real contact. Each object is then rendered under 7 lights, with 11 fixed views and 12 random views per light. Training uses 322 scenes, 1,710 objects, 12,951 counterfactual pairs, and 275,310 reference images. Annotations include a placement description.
The backbone is Qwen-Image-Edit-2511. Qwen2.5-VL and the VAE stay frozen. The DiT is fine-tuned with LoRA at rank 32 for about 16,000 steps on 8 L20X GPUs. Inference uses 40 steps and a CFG scale of 4.0. Box guidance adds 4,000 steps and an extra reference with the target box drawn on it.
Geometry-Guided Adaptation adds a third token stream. Frozen VGGT, which regresses camera, depth, and structure from an image, reads the empty background. An MLP maps those features into the DiT, plus a geometry-modality embedding. At each block, geometry, image, and text compute their own queries, keys, and values and attend jointly. Background geometry describes the scene before insertion. VGGT features of the with-object target are a stop-gradient training target and are never fed into the generator. A readout on blocks 14, 29, 44, and 59 matches them with Smooth L1 plus cosine distance. At inference the third stream stays on and evolves from the background prior.
2D supervision localizes the change. The ground-truth mask is downsampled to an occupancy map. A smoothed pixel difference between the with-object and empty images, with the object and a boundary ring removed, becomes a soft label for outside effects. It marks visible change and does not separate shadow from reflection. A spatial head on blocks 35, 47, and 56 predicts occupancy and effects with balanced cross-entropy plus Dice, weighted 0.65 and 0.35. Attention from the placement phrase is pulled toward the object region with a KL loss. The full background stays in the input. Inference does not need a mask.
Beside flow matching, the geometry, counterfactual, and grounding terms are weighted 0.02, 0.01, and 0.005.
PhysInsert-Bench has 400 cases from 103 scenes held out of training: 180 optical cases (154 same-view, 26 cross-view) and 220 structural cases with an independent reference view. Forty more cases come from DreamBooth. Fifty in-the-wild cases use ABO, NAVI, ORIDa, and references rendered from SAM 3D reconstructions.
Text guidance, one reference:
| Method | DIoU | Mask IoU | DINO-Obj. | PSNR | FID | Phy-VLM |
| Nano Banana Pro | 54.58 | 48.34 | 0.7023 | 22.85 | 46.55 | 3.48±0.09 |
| GPT-Image-2 | 38.02 | 31.92 | 0.6746 | 19.39 | 53.46 | 3.17±0.11 |
| Qwen-Image-Edit-2511 | 48.32 | 44.14 | 0.6786 | 19.34 | 60.99 | 2.94±0.15 |
| PhysInsert | 62.99 | 56.08 | 0.7276 | 24.40 | 41.96 | 3.52±0.11 |
Insert-Anything fine-tuned on InsertWorld for the same budget scores 22.66 text DIoU. OmniPaint under that setup scores 37.69. Object-region AbsRel falls from 0.2207 to 0.1195, and δ1.25 rises from 73.35% to 86.56%. LPIPS falls from 0.2260 to 0.1118.
With a box, Mask IoU 68.62, DINO-Obj. 0.7229, PSNR 25.81, and FID 37.50 are the best entries in that table. DIRECT, given ground-truth pose fitting, still leads SSIM, 0.8957 against 0.8876. The same data and budget also lift Insert-Anything and OmniPaint on the box-guided metrics.
In the wild, one reference gives PSNR 20.36 and FID 95.67. DINO-Obj. 0.7784 leads the open-source entries and trails Nano Banana Pro at 0.7881. Three references give PSNR 20.60 and FID 87.90. On DreamBooth with one reference, PSNR is 17.45, FID is 83.68, and DINO-Obj. is 0.8856, ahead of every listed system, including GPT-Image-2 at 0.8821. FID is nearly tied with Nano Banana Pro at 83.74.
Twenty-seven people scored 25 blinded cases, RealBokeh backgrounds with Distinctions-646 objects. The overall score is 4.04. Lighting, perspective, and pose rank first. Identity and background preservation stay competitive. InsertWorld fine-tuning alone moves PSNR from 19.34 to 22.53 and DIoU from 48.32% to 60.19%. Dropping GGA, spatial grounding, or the counterfactual loss leaves PSNR at 23.83, 23.73, and 23.90. The full model is 24.40.
The setting is a reference photo, a scene, and either a sentence or a box, with lighting, shadows, and perspective supposed to follow the scene. The background is not masked out first, and inference does not need a 3D proxy fitted to a pose. Text can turn a laptop, flip a spout, or place the object behind a chair or on an upper shelf.
The gain is incremental, and most of it is the data. Geometry and the 2D losses add 1.87 PSNR and 2.80 DIoU points on top of data-only fine-tuning. The gap to closed models on this bench shrinks on DreamBooth and on in-the-wild images.
At 1024 pixels, 40 steps, and CFG 4.0 on one L20X, the backbone takes about 79.4 seconds and 55.9 GiB. PhysInsert takes about 120.1 seconds and 60.8 GiB, with about 4.5% more pipeline parameters.
Identity is not a separate objective. A large gap between reference and target viewpoint still drops distinctive detail. How to combine several references is left open. The one-reference in-the-wild DINO-Obj. score already sits below Nano Banana Pro.
Phy-VLM is GPT-5.5 scoring against the ground-truth image, and the prompt says not to deduct points for size alone. The reported intervals, 3.52±0.11 and 3.48±0.09, overlap, so first place on that score does not hold. AbsRel and δ1.25 compare Depth Anything 3 estimates on the output and the ground truth, and the estimator adds its own error. Effect labels come from pixel differences, with shadow and reflection left unseparated. The main bench and the training set share one Blender counterfactual pipeline, which favors in-domain scores. DreamBooth's empty backgrounds are Nano Banana Pro removals checked by hand. The user study covers 25 cases.