ACT3D trains VLMs to actively explore 3D worlds with RL, hitting 53.2% accuracy
UCSanDiego · hf · 2026-10-19
A UCSD/Georgia Tech paper introduces ACT3D, which trains vision-language models via reinforcement learning to actively explore 3D environments from single first-person views, human-style.
- The model sees only one rendered view at a time; the 3D scene is just an interactive environment, with no multi-view or 3D representations given
- It iteratively reasons, navigates, and reorients its viewpoint before answering
- After trajectory-supervised finetuning, RL uses final-answer correctness as reward — with no direct supervision on which viewpoints to visit; the exploration policy emerges purely from performance feedback
Across six 3D benchmarks, ACT3D averages 53.2% accuracy, +22.0 points over its Qwen base and surpassing proprietary Gemini 3.6 Flash.
More from Research
- Niels Rogge launches paper feed: follow AI research by task, method, author or lab — NielsRogge · 2026-10-19
- A frozen 16MB vocabulary replaces learned embeddings in small and ternary LMs — Sai Kiran Bathula · 2026-10-19
- Meituan LongCat releases RAP: 25K paper-poster pairs as a visual coding benchmark — meituan · 2026-10-19
- BELIEFWEAVER: belief updates and world modeling for long-horizon agent failure recovery — IntelligenceLab · 2026-10-19
- Consistent Distribution Matching: data-free diffusion distillation beats DMD2 with 20GB VRAM — vector-institute · 2026-10-19
- DeepMind's XYEval: one bad user hint drops agent success from 67.4% to 36% — i_dg23 · 2026-10-19