2026-09-25
A training-free Visual Jenga method ranks object removal by SD-1.5 inpainting diversity: 91.32% pairwise on NYU-v2, 65% on HardParse, 71.43% pass on 56 full-scene decompositions.
Detectors and segmenters already name almost everything in a photo. They still do not answer a more basic question: which object can you lift first without the scene falling apart. Robot table-clearing and physically honest image editing both need support, occlusion, and co-occurrence, not class labels.
Anand Bhattad at TTIC, with Konpat Preechakul and Alexei Efros at UC Berkeley, turn that into Visual Jenga. Given one image, delete objects one at a time until only background remains. Every intermediate frame has to stay geometrically and physically well-formed. Input and output are images. A text answer such as "remove the book on the table" is disallowed: it does not localize, and language priors can fake the order.
The signal is asymmetric co-occurrence. Search "cup" and almost every result sits on a table. Search "table" and cups are rare. In probabilities, P(table | cup) is far larger than P(cup | table), so the cup depends on the table.
They estimate that quantity with counterfactual inpainting. Mask object A and let a generative model fill the hole. If the fills scatter across many object types, the slot is replaceable and A should go first. If the fills keep reconstructing the same support surface, the scene needs A and it should go last. For a cat on a table, masking the cat yields many substitutes; masking the table repeatedly yields a load-bearing surface.
The pipeline is training-free:
The diversity score is one minus mean CLIP similarity, times the DINO term, normalized by mask area. CLIP or DINO alone is close on NYU; both together matter on hard pairs. There is no 3D reconstruction and no physics engine. The bet is that a large inpainting model already stores what a well-formed scene looks like.
Pairwise tests give two masks and ask which to remove first (chance 50%). Full-scene tests run until background; a human marks the whole sequence pass or fail.
| Method | NYU-v2 pairwise (668) | HardParse pairwise (40) | Full scene (56) |
| Top-to-bottom | 59.14% | 52.25% | 41.07% |
| Small-to-large | 90.12% | 50% | 42.85% |
| Front-to-back (monocular depth) | 37.57% | 52.5% | 8.92% |
| Counterfactual Inpainting | 91.32% | 65% | 71.43% |
NYU-v2 uses Yang et al.'s indoor support graphs: 485 images, 668 unambiguous pairs, pillow-on-bed cases. Small-to-large already hits 90.12%; the method gets 91.32%, a 1.2-point gap. The other two sets do the work. HardParse is 40 internet pairs with stacks, hanging, and leaning objects, instance masks labeled by people. The 56 full scenes mix the authors' photos and web searches such as "messy desk". The best heuristic passes 42.85% of full scenes; this method passes 40 of 56.
CLIP and DINO have to travel together. Drop DINO: 89.52% NYU, 55% HardParse. Drop CLIP: 90.27% and 57.5%. Both: HardParse 65%. Sample count is equally brittle. On HardParse, N=2 or 4 is 50%, N=8 is 62.5%, N=16 is 65%. Extra samples after that barely move the needle.
The appendix wires GPT-4o (November 2024) to DALL·E, InstructPix2Pix, or Molmo+SAM+Firefly. Text orders are often sensible. Pixel outputs are not: ambiguous referring expressions, Molmo pointing at the wrong glass, editors that rewrite the room. No pairwise VLM number is reported, only failure cases.
Section 4.1 writes HardParse as 28/40 = 70%. Table 1, Table 2, and the N ablation all say 65%. The tables win.
The task is a probe: did generative models memorize support, or only names. For object erasure, scene editing, and disassembly planning, the ranker is a usable training-free baseline. Molmo, SAM 2, and SD 1.5 are off the shelf. No labels.
Read the datasets separately. On NYU the method barely beats "take the small thing first", so ordinary indoor support is already in a size prior. Stacks and leans are where it earns its keep. Statistical co-occurrence solves some dependency ranking. It does not retire a physics engine.
The authors list the obvious costs. It is slow: many inpaintings per object. It emits one order; people can hold several. If Molmo, SAM, or Firefly fails, the pipeline fails: Firefly reinserts objects when shadows are strong, Molmo misses a third glass, SAM crops a monitor to its screen. There is no explicit physics, only co-occurrence. Object grain is undefined (one sheet vs a stack), so the 71.43% full-scene pass rate is a human judgment, not a leaderboard.
Appendix E says HardParse was used to pick hyperparameters. The main text says it was never used that way. Those two sentences disagree. VLM baselines have no pairwise accuracy, so GPT-4o's text order cannot be compared to 65%. Firefly is a closed commercial API. The authors point at video generators as world simulators for P(A | do(¬B)); that experiment is not in the paper.