VibePrevisBench: 11 Coding Agents Build 4D Scenes in Blender, Best Scores 77.7 and Can't Self-Correct

VibePrevisBench: Evaluating Agentic Coding for 4D Scene Generation in Blender

Yuzhe Chen, Muxi Diao, Lele Yang, Wenbin Xu, Xuyang Liu, Qile Su, Changyi Liu, Jiankang Chen, Kaiyu Jiang, Haoyu Zhang, Hongyi Fu, Jianxiong Wang, Tianke Zhang, Haonan Fan, Yingxin Li, Bingyi Li, Xu Wang, Xuanyu Zheng, Bin Wen, Tingting Gao, Wentao Zhang, Jiale Cao

cs.CV, cs.AI

2026-11-12

VibePrevisBench has 242 tasks scored by rule checks on the 4D scene and a VLM check on the render. Of 11 coding agents, GPT-6 Astra leads at 77.7; all view their renders but fail to fix errors.

What problem this solves

Controllable video generation models can now take a graybox proxy video as input: an untextured Blender render that pins down layout, character motion, camera and cuts, while the model paints realistic appearance on top. Artists used to build these by hand, which is slow and expensive. Coding agents can now write Blender code to rebuild a 4D scene (3D plus time) and render it. What has been missing is a way to measure how well they do it.

Existing benchmarks leave three gaps. Inputs are narrow, mostly text prompts or videos to reconstruct. Content is static or short, such as single objects, static scenes or clips without people. And evaluation looks only at rendered frames, never at the scene behind them. The paper shows why that matters. In one task, riders were supposed to turn at eye level, but the weak agent filmed from above, and the VLM judge still gave it 95.5. In another, the car was in frame and turning, yet it appeared as a tiny dot. The authors report that 31% of annotated actions pass the rule checks but are hard or impossible to see in the proxy video.

Method

VibePrevisBench has 242 tasks: 94 scripts, 86 reference videos, 39 photo tasks and 23 continuations, across seven scene domains and 33 places. Most are long, multi-shot scenes around 40 seconds: 88% have two or more shots, 80% a moving camera, 79% two or more people. Each task carries hidden ground-truth (GT) facts, 99.0k checks in total.

An agent delivers a Blender Python program. The evaluator re-runs it from scratch in a clean sandbox and ignores any video the agent rendered itself, so a video cannot show something the program doesn't build. A delivery that fails any of three gates (runs standalone, renders fully, reopens in Blender) scores 0 on both checks.

Two complementary scores:

The overall score is the mean of the two. Tasks were pruned using item-rest correlation and spread across all 11 agents, not any single agent's score.

Results

AgentRule (all)Screen (all)Overall
GPT-6 Astra70.085.477.7
GPT-6.1 Sol69.083.176.1
Gemini 4 Argon68.481.775.0
Opus 5.568.479.874.1
Muse Spark 1.357.265.061.1

The top four lead the other seven by at least 3.9 points, but five of the ten adjacent pairs cannot be separated by a paired bootstrap. The ranking holds across 18 scoring variants (Kendall τ ≥ 0.85) with GPT-6 Astra always first. Only 35 of 242 tasks see any agent reach a Scene score of 90, and the per-task best of all 11 agents reaches only 80.6 overall.

Cost and score diverge. Per-task cost spans 45x, from $0.73 (GLM 5.3 Flash) to $32.59 (GPT-6 Astra). GPT-6.1 Sol gets 98% of the leader's score for 21% of the cost; DeepSeek V4.1 Flash gets 90% for 6%.

Three observations:

Why it matters

For anyone building a coding agent plus graybox plus video model pipeline, the paper gives a concrete list of shared weak spots: camera height, camera move type, shot count, and props mentioned only in text. A stronger model alleviates but does not fix them.

Two practical takeaways. First, add explicit item-by-item checks of renders against the brief, especially for camera height, move type and shot count. Second, Blender API fluency is not the bottleneck: the three weakest agents often call an API removed in Blender 5, but later rounds repair it and runs with failed calls score within 0.1 points of those without. On model choice, GPT-6.1 Sol or DeepSeek V4.1 Flash offer far better value than the top-priced model.

The benchmark design is also reusable: a rule check on the scene and a visual judge that only sees items the rules passed.

Limitations

The authors acknowledge that some GT facts come from model-based estimation pipelines, and that the physical-validity check barely notices floating people (lifting them 1 m costs 0.8 points). Several evaluated models also helped build tasks. The paper measures a 1.3-point own-family preference in the Gemini 4 judge with no rank change, and says a small preference among the Gemini models on media tasks they helped annotate cannot be ruled out.

A few further doubts. The claim that agents fall back to common choices because challenging requests are rare in training data is a hypothesis; the paper does not test it against training data. The top four are also the newest and priciest models, so the tiers do not isolate which capability matters. Within-family comparisons cover only two pairs, one of which also changes tier. The "looking more doesn't help" ablation is one agent on eight script tasks, and the oracle-caption test is one agent on five tasks. Each task is run once. And whether better proxy videos yield better final real-world videos is shown only qualitatively.

Terms

Source

Related papers

All paper explainers