Scene-Designer: Learning Design Intelligence for Agentic 3D Scene Generation
Xuanyu Liu, Hongyan Feng, Miao Pan, Sunlai Chen, Lizhou Liang, Bing Sun, Wenqi Zhang, Xuhong Zhang
cs.CV
2026-10-16
ZJU fine-tunes Qwen3-VL-8B to design and critique indoor scenes: four-scene ranking reaches 76% vs 15% for GPT-5.2, and five-plus-step orientation fixes drop from 40% to 7%.
A usable indoor scene is a design job. The model has to fix an obvious mistake in one move, and it has to choose which of two physically valid layouts a person would actually live in. Zhejiang University's OmniAI lab calls that pair of skills design intelligence.
Hooking a general model up to tools leaves the experience gap in place. LayoutGPT, Holodeck, I-Design, and SceneWeaver all build iteratively, and the decisions still come from a general-purpose model. 3D-FRONT stores static object poses, with no record of why a placement changed. On 100 scenes from GPT-5.2 inside the SceneSmith scaffold, physical errors are 60.7% of construction errors and semantic or layout errors are 39.3%. Collisions, boundary violations, and blocked doors show up in about nine out of ten trajectories. A blocked window is usually one relocation, yet about 18% of those cases take more than five moves. Interaction-orientation errors, such as a chair facing away from the table, are fixed in one move only 7.5% of the time, and 62% take at least five. Among assets, 34% are placed once and left alone, and 24% are edited five or more times.
Critique is thin on the same data. GPT-5.2 scores about 72% on pairwise comparisons and 15% when it must rank four scenes in full. Base Qwen3-VL-8B-Instruct scores 2% on that four-scene ranking, near chance.
Scene-OTA-35K is the design trace. GPT-5.2 builds with SceneSmith, and a manual filter keeps the strong results: 1K scenes and about 35K Observation-Thought-Action steps. Causal Trajectory Consolidation flags a step that neither moves positions and orientations closer to the final layout nor reduces the remaining error count. GPT-5.2 then marks the suspect step as a useless trial, a useful correction, or uncertain. Only high-confidence useless steps are deleted, and the short trace is replayed so the final scene stays the same. Average length drops from about 110 steps to about 30, and each kept step gets a written rationale.
Scene-OTC-15K holds about 15K critique examples. Good scenes are degraded in three grades: mild edits hurt layout coherence only, a middle grade breaks functional relations, and a severe grade adds collisions and out-of-bounds placements. Perturbations can stack, hit the same object harder, or land on different objects. The first two schedules rank themselves by severity. The third needs a human label, because the same nudge matters more on some furniture than others. Held-out tests are 94 pairs and 55 four-scene rankings.
The base model is Qwen3-VL-8B-Instruct, updated in full. Joint supervised fine-tuning on both datasets runs for 2 epochs at learning rate 1e-5 on 4 A800 GPUs and yields Scene-Builder.
Scene-GRPO splits Scene-Builder into a frozen Critic and a trainable policy. GRPO, group relative policy optimization, samples 4 actions from the same parent scene. The Critic ranks the siblings, then compares each child with its parent: +0.5 if the edit helps, -0.5 if it hurts, and -0.5 if the branch fails to render. Trajectory-level terms pay for length and object count. Full credit needs at most 25 steps and at least 25 objects. The length term is zero at 45 steps, and the object term is zero below 10 objects. The learning rate is 1e-6, trajectories cap at 80 steps, and the top-ranked branch continues.
At test time the same four-way expansion stays on. The Critic picks one branch and the weights stay fixed.
LLM-based methods share the prompt "Design me a {roomtype}" and generate 10 scenes per room type. GPT-4 scores realism, functionality, layout, and completeness from 0 to 10. Object count is treated as higher-is-better; out-of-bounds objects and collisions are lower-is-better.
Scene-Builder already hits zero collisions and zero out-of-bounds objects in bedrooms and living rooms, level with SceneWeaver and I-Design. Holodeck bedrooms average 32.2 objects and 38.5 collisions. ATISS, DiffuScene, and PhyScene place only 3.3 to 3.9 bedroom objects, with completeness from 3.6 to 4.2.
| Method | Room | Objects | Collisions | Realism | Function | Layout | Completeness |
| SceneWeaver | Bedroom | 14.0 | 0 | 9.2 | 9.8 | 8.4 | 9.4 |
| Scene-Builder | Bedroom | 19.6 | 0 | 8.8 | 9.2 | 8.4 | 8.9 |
| Scene-Designer | Bedroom | 20.0 | 0 | 9.5 | 9.5 | 8.8 | 9.4 |
| Plus test-time Critic | Bedroom | 20.8 | 0 | 9.8 | 9.8 | 9.2 | 9.7 |
| SceneWeaver | Living room | 17.3 | 0 | 9.1 | 9.5 | 8.0 | 8.7 |
| Scene-Builder | Living room | 17.9 | 0 | 8.0 | 8.6 | 8.2 | 7.8 |
| Scene-Designer | Living room | 20.3 | 0 | 9.6 | 9.0 | 9.3 | 9.0 |
| Plus test-time Critic | Living room | 21.1 | 0 | 10.0 | 9.4 | 9.6 | 9.5 |
In living rooms, realism moves from 8.0 to 9.6 and layout from 8.2 to 9.3. Test-time Critic guidance pushes realism to 10.0. Bedroom function is the exception to a clean sweep: SceneWeaver scores 9.8, Scene-Designer 9.5, and the test-time Critic ties 9.8.
Bathrooms and offices are also collision-free. SceneWeaver's functionality score is 10.0, and its offices average 40.0 objects. With the test-time Critic, bathrooms land at 22.0 objects, realism 9.7, function 9.5, layout 9.1, completeness 9.7. Offices land at 24.2, 9.8, 9.4, 9.2, and 9.6. SceneWeaver's office realism, layout, and completeness are 9.0, 8.0, and 8.7, so function and object count still favor SceneWeaver.
Repair length is the GRPO signal. Versus Scene-Builder, the share of cases that still need five or more moves falls from 40% to 7% for interaction orientation, 21% to 5% for blocked windows, 50% to 23% for scale mismatch, and only 15% to 13% for door clearance.
| Model | Pairs (94) | Four scenes (55) |
| Qwen3-VL-8B-Instruct | 72% | 2% |
| GPT-5.2 | 72% | 15% |
| OTC only | 86% | 51% |
| OTA+OTC | 89% | 76% |
Joint training lifts four-scene ranking from 51% to 76%, with Top-1 at 82%. Across five shuffles of image order, pair accuracy is 90.00±2.07% and four-scene accuracy is 75.64±6.88%.
The design policy is Qwen3-VL-8B. SceneWeaver and Holodeck still ask a general model for each edit, and those traces run long. Scene-Designer first imitates traces with the useless steps removed, then lets a frozen Critic choose the branch. On living rooms that raises realism from 8.0 to 9.6 and layout from 8.2 to 9.3, and four-branch test-time selection goes on to 10.0 and 9.6.
The recipe travels further than any single column. Compress traces before imitation, build critique labels with graded perturbations, and split the reward into a sibling ranking plus a parent-child check. Collisions are already zero after supervised fine-tuning. GRPO buys functional grouping and shorter repairs. Against SceneWeaver, the gains sit in some columns and miss others.
The paper calls the link between design and critique a trend. Adding design traces moves four-scene ranking from 51% to 76%. The reverse direction is two qualitative cases, a dining room and a reading corner. The main tables do not report an OTA-only policy.
All four visual scores are GPT-4 ratings from 0 to 10, and the main text has no human preference study. That sits awkwardly next to the claim that general models rank scenes poorly. Object count is defined so that more is better. SceneWeaver offices average 40.0 objects and function 10.0, while Scene-Designer with the Critic averages 24.2 objects and function 9.4.
Each room type contributes 10 scenes. Most critique ranks are fixed by how hard the perturbation was, so the Critic may be detecting edit artifacts. The share of cross-object comparisons that needed a human label is unreported. Raw traces and the deletion of useless steps both pass through GPT-5.2, and the main table never scores that teacher. The Critic stays frozen during GRPO, so critique does not improve inside the RL loop. Rooms stop at bedrooms, living rooms, bathrooms, and offices.