NVIDIA's GATOR: generative 3D model + GPT-6 agent turns casual photos into simulation-ready objects

GATOR: Generative and Agentic 3D Object Reconstruction From Casual Images

Qirui Wu, Stan Birchfield, Hesam Rabeti, Angel X. Chang, Bowen Wen

cs.CV, cs.RO

2026-10-08

GATOR drafts posed, textured 3D objects from cluttered photos with a generative cascade, then lets a GPT-6 agent refine them, halving Pixal3D's chamfer distance on real tabletops.

What problem this solves

Turning casual photos into 3D assets that are actually usable: complete geometry, real textures, and the pose to place each object back into its scene. "Casual" is doing specific work here. No object isolation, no calibrated capture, no planned camera path. The target sits in tabletop clutter or a room corner, partly occluded, seen from a few irregular viewpoints, and cameras, depth, and masks all have to be estimated from the images.

The two existing paths each stall halfway. Generative reconstructors (LRM, TRELLIS.2, SAM 3D) fill unobserved backsides from learned priors, so a plausible-looking completion can diverge from the real object; when image, mask, and geometry arrive as independent conditioning streams, the denoiser has to infer how they correspond. Pure agentic pipelines, where GPT-6-Astra writes Blender code directly, control structure and materials but burn API time and cost building from scratch, and fine geometry suffers. In Figure 1 the agent misplaces a strainer's ear and simplifies its handle into crude shapes. GATOR chains the two: a generative model drafts a complete, scene-aligned asset, and an agent makes targeted repairs on top of it.

Method

The generative stage rides on TRELLIS.2's structure-geometry-appearance cascade:

• N views (any count) are encoded on a shared patch grid: frozen DINOv3 features for RGB and the target mask, a trainable encoder for the pointmap, the per-pixel 3D coordinates unprojected from depth.
• The central piece is a local modality mixer: before any cross-view attention, the RGB, mask, and pointmap tokens within each patch attend to one another. The three modalities are already aligned on the patch, so the correspondence gets built into the fusion instead of being left for the denoiser to rediscover, and reading the three cues together lets the model cross-check estimated depth against appearance and mask, which pays off when upstream estimates are noisy. RGB and pointmap keep scene context for completing occluded parts; mask tokens mark the target. The structure stage sees the full crop to reason about occlusion, while geometry and appearance project features only from target-masked images so occluders never leak into texture.
• Text conditions every stage: a SigLIP2 embedding of the category name globally modulates structure generation, and T5Gemma2-encoded captions enter geometry and appearance as cross-attention tokens. When views are sparse and several completions are plausible, category semantics pull toward the consistent one.
• Pose: normalization translates and scales but never rotates, so outputs keep scene-relative orientation; differentiable rendering of silhouettes and depth then refines translation and scale.
• Training is synthetic-only: about 980k assets composed into procedurally cluttered scenes (the S3 pipeline), with clean observations mixed with perturbed or image-estimated cameras and depth to mimic casual capture.

The agentic stage hands the generated asset, the input images, and Blender to GPT-6-Astra (Codex CLI, extra-high reasoning effort). The agent decides what to fix: completing missing structure, repairing topology, correcting UVs and materials, including the legibility of printed text and logos. After each edit it renders the original and candidate under matched cameras and lighting and reverts anything that degrades agreement with the observations, keeping the best inspected asset within a fixed budget (ten minutes per object in the main experiments). The generative model is never retrained.

Results

On ScanNet++, where cameras, depth, and masks are all estimated (SAM3 does the segmentation), GATOR cuts CD by 23% against runner-up RecGen and beats from-scratch GPT-6-Astra on every metric. The ablation adds one component at a time on 55 HANDAL objects: the modality mixer mostly drops ADD-SB from 0.057 to 0.037, which points to robustness against estimation error; text conditioning lifts all nine metrics (F1 0.872 to 0.930); agent refinement mainly helps appearance (PSNR 18.93 to 19.60).

The efficiency numbers are the most striking. Generation alone takes about 0.5 minutes (roughly 32 s of generative compute) and already outperforms GPT-6-Astra given a 20-minute budget. From scratch, GPT-6-Astra returns valid assets for 0 of 10 objects at one minute, 5 of 10 at two minutes, and all ten only from five minutes onward. More views keep helping: with two views, GATOR's CD is about half of Pixal3D's with eight.

Why it matters

For robot simulation, scene reconstruction, and digital-asset work, casual photos now yield simulation-ready objects with PBR materials and scene-relative pose, with no isolated or calibrated capture. Generalization to four real datasets happens with zero fine-tuning, resting on a synthetic-data-plus-error-injection recipe that is portable in itself.

Methodologically this is a clean case of "generative draft, agentic repair": the agent never models from scratch, it makes local edits anchored to instance-level geometry and pose, with time budgets, reverts, and no retraining of the generator. That division of labor should transfer to other 3D generation tasks. Honesty check: on clean synthetic data the margin over the strongest baseline is incremental (19.7% CD reduction on Toys4K). The real gaps open up in cluttered real scenes (tabletop CD halved) and in pose recovery, a dimension several baselines simply do not output.

Limitations

• Agentic refinement barely moves average geometry metrics and slightly worsens CD, F1, and IoU on HANDAL. The authors argue the edits are local and benefit simulation in ways the metrics miss; the quantitative simulation study sits in the supplementary material.
• The pipeline leans on external estimators: Pi3X-style camera and depth estimation, SAM3 masks on ScanNet++. The mixer tolerates that noise rather than removing it, and because normalization never rotates, metric scale requires metrically scaled input depth.
• The refinement agent runs on a commercial API (GPT-6-Astra). Per-object cost for the ten-minute budget is not reported, and reproducibility is limited; the from-scratch GPT-6-Astra baseline is flagged as fully proprietary and excluded from best-marking in the tables.
• The ablation covers only 55 objects. On tabletop pose recall, GPT-6-Astra scores [email protected] of 100% versus GATOR's 98.47%, so the proprietary model still wins that one cell.

Terms

• pointmap: per-pixel 3D coordinates unprojected from depth, giving the model direct geometric evidence.
• modality mixer: GATOR's core module, where RGB, mask, and pointmap tokens attend to one another within each patch.
• PBR materials: physically based rendering materials (albedo, roughness, metalness maps) that simulation engines consume directly.
• TRELLIS.2: a 3D generator with a structure-geometry-appearance cascade; GATOR's structure denoiser initializes from its 1.3B checkpoint.
• ADD-SB: a bidirectional surface distance scoring shape and pose jointly; @0.1 is recall at 10% of the reference diameter.
• chamfer distance (CD): mean nearest-neighbor distance between two meshes, normalized by reference diameter here.

Source

• arXiv 2610.11215v1
• PDF
• Hugging Face

What people are saying

• NVIDIA's GATOR turns casual photos into simulation-ready 3D objects with agentic refinement AjayMandlekar

Related papers

• One Model Maps Layout-Free Dirty Markers to SMPL-H via 113 Anchors
• One model unifies four 3D part segmentation tasks, beating specialized baselines by up to 34.5 mIoU
• Text scores tie model pairs near 50/50; structural scores split them 74.2 to 25.8
• Lag-chosen gain cuts unguided SiT-XL ImageNet FID from 28.0 to 12.2 at NFE 25
• StreamMAE: self-supervised pretraining on 95 hours of continuous video matches i.i.d. training
• VibeEdit edits images from marks drawn on the canvas, beating text-prompted baselines 79.9 vs 67.4

All paper explainers