NVIDIA's GATOR turns casual photos into simulation-ready 3D objects with agentic refinement
AjayMandlekar · x · 2026-10-11
NVIDIA researchers introduced GATOR, a generative + agentic 3D object reconstruction pipeline that converts casually captured photos into simulation-ready assets—demonstrated with humanoids interacting with reconstructed furniture in physics sims like water pouring.
Key components:
- Pose-aware modality mixing: local attention fuses patch-aligned RGB, target-mask, and pointmap features before cross-view reasoning, enabling complete object reconstruction in clutter with scene-relative pose;
- Text-guided generation: stage-specific adapters use category names for sparse structure and object descriptions for geometry/appearance, recovering detailed textures and physically based materials;
- Agentic refinement: an agent runs an edit-render-review loop in Blender, using original images as evidence to fix structure, topology, and textures while preserving reliable geometry and pose—improving sim readiness without retraining the generator.
Paper is out; code coming soon.
More from Multimodal
- ZPix Adds Qwen-Image 2.1 Turbo with Checkerboard-Fix VAE for Local Image Generation — SamuelTallet · 2026-10-11
- Krea2 Turbo 2-Step Distill LoRA Final: 4x Faster, Detail Within 92% of the 8-Step Teacher — TimeTruth2490 · 2026-10-11
- CUT debuts: build scenes with AI characters, voices, and a timeline — rileygstaliger · 2026-10-11
- Kling AI 4 Flash demo: alien short at 21:9, 720p — Forsaken_Stuff_Ai · 2026-10-11
- AI-built 3D scene demo: TripoSplat assets, Claude Opus compositing on PlayCanvas — willeastcott · 2026-10-11
- Open-source MCP server lets Claude Opus 5.5 write complete FL Studio tracks end to end — delarein · 2026-10-11