NVIDIA's Foundation4D unifies AV perception and reconstruction, topping Waymo and nuScenes
nvidia · hf · 2026-10-23
NVIDIA introduces Foundation4D, a unified feed-forward model that jointly estimates geometry, performs 3D detection, predicts semantic occupancy, and reconstructs Gaussians from multi-camera driving video.
- Architecture: a shared backbone with factorized spatiotemporal attention fuses observations across cameras and time; the video pipeline uses pose graph optimization to refine camera trajectories and assemble object tracks across windows.
- Data: ships an annotation pipeline and releases the PAI-Foundation4D benchmark with dense depth and occupancy labels for 1,253 driving clips.
- Results: lowest depth AbsRel on Waymo and nuScenes, highest detection mAP on nuScenes, highest occupancy mIoU on OpenOccupancy. Co-training helps: detection+geometry+occupancy adds 1.2 top-class mAP over detection-only, and geometry+Detection cuts camera-to-rig translation error by 13.4% relative.
More from Research
- SlopCodeBench: Top Model Scores Just 16% on Long-Horizon Coding Agent Benchmark — ajratner · 2026-10-23
- Charter v0.3: a formally verified language for institutional rules with AI constitutional prompts — QuixiAI · 2026-10-23
- QuixiAI hails eFLINT, will redesign Charter as a superset of the formal rule language — QuixiAI · 2026-10-23
- ArchitectureIQ Benchmarks LLMs' Training Intuition: GPT-5.6 Sol Hits 76.4% vs 33% Chance — ZimingLiu11 · 2026-10-23
- Claude Links Groundwater Pulse Pumping to the Sleeping Brain's Waste Clearance — abstrusejoker · 2026-10-23
- 345K trajectories reveal LLM agent personalities: Claude most planful, GPT-5.5 least energetic — mdredze · 2026-10-23