Thinking in Blender: Staged Executable Inverse Graphics with Vision-Language Models
Guangzhao He, Rundong Luo, Wei-Chiu Ma, Hadar Averbuch-Elor
cs.CV
2026-06-02
SEIG turns one image into staged Blender code with Claude Opus 4.7. On NeRF synthetic it reaches PSNR 13.58 and wins 5 of 6 metrics vs VIGA, including the SAM-3D variant.
Recovering an editable 3D scene from one image is an old inverse-graphics problem, and it is badly underconstrained. Geometry, materials, layout, and lighting trade off against each other. Fixing one factor can hide errors in another. Neural fields such as NeRF and 3D Gaussian Splatting match novel views well, but they bury those factors in latent weights. You cannot open the result in Blender, swap a lamp, or pick up a mug.
A newer line of work lets vision-language models write scene programs. VIGA already frames the task as a write-render-compare-revise loop, yet it still treats all scene factors as one joint optimization. This Cornell paper asks a narrower question. With no SAM, no SAM-3D, no differentiable renderer, and no multi-view supervision, can an off-the-shelf VLM reconstruct a single image as an executable Blender file.
SEIG copies the order a Blender artist already uses, and it commits each stage before the next one starts.
Scene graph and initialization. The VLM decomposes the image into a hierarchy whose leaves must be Blender primitives such as spheres, cubes, and cones. Initialization builds a coarse scaffold from those text attributes and places a rough camera and lights. The point is stable object names, not pixel match. Missed objects are hard to recover later, so the pipeline samples multiple independent graphs and scaffolds, then a selector keeps the most complete, plausible one. The figure caption says four rollouts.
Geometry. Per named object, the agent may edit meshes and curves, apply transforms, add missing parts, and tidy hierarchies. It can render other views, isolate an object, and revert a bad edit.
Materials. Placeholder flat colors become Blender PBR shaders, which approximate real surfaces with roughness, metallic, and normal maps instead of a baked texture. A material-only tool blocks geometry edits.
Composition. Only transforms, scale, contact, and optionally the target camera. Geometry and materials stay frozen.
Lighting. Shape, appearance, layout, and camera stay frozen. The agent tunes light type, pose, energy, color, softness, plus exposure and color management, and it is told to revert over-dark or over-exposed renders.
Inside each stage a generator writes and executes code, then a verifier inspects only that stage's factor and returns an actionable checklist. Round budgets are 5, 3, 3, and 2 for geometry, material, composition, and lighting. If the budget runs out, the verifier must pick the best attempt and move on. The backbone is Claude Opus 4.7 for every role, with no fine-tuning.
Evaluation uses five renders from each of 7 NeRF synthetic scenes. The eighth scene, materials, is dropped because of specular metal spheres. A second set of 13 object-centric scenes from VoxHammer is labeled Edit3D. The baseline is VIGA on the same VLM, in two flavors: the original pipeline with SAM and SAM-3D pre-reconstruction, and a VLM-only ablation. When a reference mesh exists, reconstructions are registered with Neural Deformation Pyramid and ICP. The smaller Chamfer is kept, and metrics are computed from the reference camera so camera error does not leak into the score.
| Method | NeRF PSNR | NeRF DINO | NeRF CLIP | Edit3D PSNR | Edit3D DINO |
| VIGA VLM-only | 12.33 | 0.6221 | 0.8451 | 11.52 | 0.5606 |
| VIGA full | 11.18 | 0.5545 | 0.7986 | 12.48 | 0.4832 |
| SEIG | 13.58 | 0.7188 | 0.8830 | 12.65 | 0.6293 |
SEIG wins five of six metrics on both sets. On NeRF, PSNR is 1.25 above VLM-only VIGA and 2.40 above the SAM-3D variant. DINO rises from 0.6221 / 0.5545 to 0.7188. The loss is SSIM: 0.6881 versus 0.7122 on NeRF, 0.6737 versus 0.6776 on Edit3D. VIGA full is worse than VLM-only on NeRF.
Qualitative examples of VIGA full show fragmented, mis-colored meshes, and a humanoid with a Janus face copied onto the back of the head. SEIG builds the figure from primitives and avoids that failure. On a mostly occluded bread basket it invents round loaves instead of bread sticks: consistent with the silhouette, wrong as geometry. Both VIGA variants fail to assemble even a coherent basket on the same image.
Because lights, materials, and objects are named Blender entities, relighting, part duplication, texture edits, rearrangement, and rigid or soft-body simulation run on the file with no extra training. The physics examples attach Blender modifiers and do not remesh.
For people building 3D agent stacks, the result is a harness claim. Task decomposition moved the needle more than specialist tools. The same Opus 4.7, staged, beats a pipeline that lifts meshes with SAM-3D first. Those specialist meshes can be overwritten or shattered by the later VLM loop. The output is a graphics program, so relighting and physics are file operations rather than a second research project in factorized NeRFs.
It does not replace neural reconstruction. PSNR 13.58 is still poor in absolute terms, and the assets are primitive assemblies. Use this when you need an editable scene graph. Do not use it when you need photographic novel views. The fair comparison is VIGA-style executable inverse graphics, not Instant-NGP.
The authors flag two issues. The pipeline is greedy: a bad geometry commit poisons later stages, and a global revisit would multiply API cost. Repeated generator-verifier calls are also expensive. The paper gives no wall-clock or dollar numbers.
The test set is small, about 35 synthetic images and 13 object scenes, with no variance and no per-stage ablation, so which stage actually pays remains an inference. SSIM is worse than the simpler VLM-only baseline on both sets. Staging is not a free lunch for every metric. Single-view ambiguity remains, as the bread-basket case shows. The tools are tied to Blender's Python API.