NVIDIA Research releases Foundation4D: unified 4D perception model turns camera video into 3D scenes, no lidar needed
Scobleizer · x · 2026-10-28
NVIDIA Research unveiled Foundation4D, a unified 4D perception and reconstruction foundation model for autonomous driving that converts multi-camera driving video into geometry, 3D objects, semantic occupancy, and Gaussian splat reconstructions.
- A shared backbone fuses information across cameras and time; video processing refines camera trajectories and links objects into tracks
- When used as priors for NVIDIA Omniverse NuRec reconstruction, it enables dataset annotation prep without lidar input, greatly simplifying training data production
- The team also released the PAI-Foundation4D benchmark with dense depth and occupancy annotations for 1,253 driving clips
- Leading benchmark results: Waymo depth AbsRel of 0.09 (δ₁.₂₅ 91.97%) and 0.15 on nuScenes; camera trajectory accuracy (ATE) substantially outperforms DVGT-1 and other baselines
Project page, white paper, and dataset are all publicly available.
More from Embodied
- Houston Robotaxi called most 'fake' AV service map: 40 min to cross 2 blocks — reed · 2026-10-28
- Indie roboticist gets first anthropometry-based robot walking with agent-driven design loop — IanPritchard · 2026-10-28
- Reward AI's OM-1 Learns Manipulation Skills From Human Demonstrations for Cross-Body Transfer — CyberRobooo · 2026-10-28
- 4 Stanford freshmen build assistive walker robot in 36 hours, win grand prize at CalHacks 13.0 — Scobleizer · 2026-10-28
- Just Action Model: decoupling what separates VLA from world-action models — Yonghao Yu · 2026-10-28
- NVIDIA's ThinkWorld learns robot skills from unlabeled human videos via chain-of-dynamics reasoning — nvidia · 2026-10-28