VibeWorlding: Can Multimodal Agents Construct 3D Open Worlds End-to-End?
Yansong Ning, Jingwen Ye, Zhongkai Wu, Yang Sun, Yiqin Zhu, Xingyi Li, Weidong Zhang, Hao Liu
cs.AI
2026-08-15
HKUST(GZ) and Tencent build VWE-Bench (6,828 queries) plus an RL gym for 3D world-building agents: GPT-5.5 reaches only 57.3% overall Pass@1 with collision-free 3D editing as the bottleneck, while a GRPO post-trained open 30B-A3B hits 59.3% and beats every closed frontier tested.
Building an interactive 3D world from one loose sentence ("an eerie wasteland shrine") is a shared want across games, simulation, and embodied AI. Prior systems either pipeline the job through fixed stages (SceneCraft, 3D-GPT) or let one multimodal model drive tools over multiple turns (SceneWeaver, SAGE). Both lines share a flaw: evaluation happens on idealized, simple queries, so nobody could say systematically where current models actually break. And with no open framework, whether training could close the gap was untestable.
The paper answers with three artifacts: VWE-Bench, an RL gym with a verifier, and two open post-trained models.
VWE-Bench is built world-first. Artists assembled 323 seed worlds from 2,616 assets; an MLLM read each world and reverse-wrote the user request that would produce it; humans filtered the results into 6,828 queries. Two families:
Every query is scored on six dimensions: physical feasibility (collision, height) plus intent fulfillment (ecological plausibility, 3D understanding, 3D reasoning, asset retrieval).
VibeWorlding-Gym exposes the environment as five MCP tools (retrieve, add, delete, rotate, translate) plus a Blender service that returns five 1280×720 renders per turn as multimodal feedback. The dual-constraint verifier (geometric collision checks by rule, intent by Gemini 3.5-flash against rubrics) serves as both judge and RL reward source. Training is two-stage: 2 epochs of cold-start SFT on filtered trajectories, then 1 epoch of GRPO with outcome-only reward (a [0,1] score from the proportion of correctly modified assets on Verified queries, binary pass on Unverified). No intermediate shaping, deliberately, to avoid reward hacking. The 8B model trains on one node of 8×H20; the 30B-A3B on three nodes.
| Model | Verified | Unverified | Overall Pass@1 | Human blind pass rate |
| GPT-5.5 | 60.4 | 55.7 | 57.3 | 33.3 |
| Qwen3.8-Max | 58.1 | 62.3 | 56.9 | 46.7 |
| Gemini 3.5-flash | 44.4 | 47.5 | 42.7 | 13.3 |
| Claude-Opus-4.8 | 42.9 | 23.0 | 24.3 | 13.3 |
| Qwen3-VL-30B-A3B (base) | 22.0 | 13.3 | 13.6 | 13.3 |
| VibeWorlder-30B-A3B (ours) | 64.5 | 62.3 | 59.3 | 47.2 |
Three findings. One: frontier models are far from solving the task; the best two sit below 60% overall, Claude-Opus-4.8 manages 24.3%, and existing agent scaffolds (SceneWeaver, SAGE) on GPT-5.5 reach only 13–16%, so the missing piece is 3D capability in the model, not workflow orchestration. Two: the bottleneck is collision-free precise editing. Collision is the weakest dimension for every model at 59–68% even after RL; a representative failure moves an object exactly 77 m in precisely the opposite direction. Three: RL fixes part of it. The 30B base climbs from 13.6% to 59.3%, overtaking every closed model tested. SFT mostly establishes physical and ecological competence; 3D understanding rises 0.17 to 0.80 and 3D reasoning 0.04 to 0.69, the latter still the weakest dimension after RL — only partially unlocked.
The verifier itself holds up: at the system level its rankings correlate with human blind evaluation at Spearman ρ = 0.88, with 83.4% case-level agreement on holistic Pass@1.
For agent-training groups this is the first complete open stack carrying the verifiable-reward-plus-multimodal-feedback recipe into 3D: benchmark, sandbox, verifier, SFT, and RL are all reproducible and portable to other tool-use domains. For content pipelines the practical read is that frontier models are not yet usable zero-shot, but an 8B open model after RL reaches 41.4% and matches Gemini 3.1-pro, so the entry bar for an in-house world-building assistant sits lower than assumed. For MLLM researchers, the diagnosis "understands the world, edits it inaccurately" pins the next target on quantitative spatial reasoning.
The 59.3-versus-57.3 margin over GPT-5.5 is thin, and the open models trained on this benchmark's own distribution hold a home-field edge; the claim that closed frontiers all sit below 60% is on firmer ground than the open-model win.
The authors list four: the sandbox offers only five atomic operations with no high-level tools like "instantiate a football field"; the most complex world has just 258 assets; outcome-based reward is inherently sparse for long-horizon building; the asset library is cartoon-styled only, and the taxonomy omits image- and video-to-world settings.
Two more concerns from a close read. Verified Pass@1 averages a proportional score, not a strict pass rate, so its column is not directly comparable to the Unverified column. And the reward verifier, Gemini 3.5-flash, is also a competitor; the human-agreement checks offset this, but Unverified-subtask rankings set by that judge's tastes (Qwen3.8-Max topping that column at 62.3%) carry residual risk of scoring bias.