BrickWright: Agentic Creation of Physically Feasible Brick Builds
Ava Pun, Kevin Dai, Sally Chen, Maxwell Jones, Ruixuan Liu, Kangle Deng, Changliu Liu, Maneesh Agrawala, Deva Ramanan, Jun-Yan Zhu
cs.CV
2026-10-23
A LEGO Python SDK plus a physical validator let pre-trained agents check and repair their own builds; all four agents hit 100% valid output on 512 prompts, up from 8.6–69.3%.
Coding agents are already good at digital 3D modeling; with no fine-tuning they produce articulated objects and full scenes. Physical assembly is where they fail. A LEGO build has hard constraints: every part must come from the real catalog, parts cannot interpenetrate or float, and the result should hold together as one piece. Asked to build directly in LDraw, Codex with GPT-6 Astra turns out a valid build 69.3% of the time, and lightweight GPT-6 Luna only 8.6%. The failures are mesh intersections, floating bricks, and builds that shatter into separate chunks.
Effort is not the problem. Astra under Codex writes its own generation and validation scripts in over 80% of runs; hand-rolled checks simply do not capture connector-level physics. The other line of work, specialized generators such as BrickGPT and BrickNet, trains physics constraints into the network but generalizes worse and follows prompts less faithfully than commercial agents.
The design idea merges code-as-3D-model agents with the regularity of bricks: every part is a geometry with typed, localized connectors, and how a stud mates with an antistud is a rule you can write down. No fine-tuning is needed, just a good toolkit and a validator the agent can run.
That toolkit is BrickLayer, a Python package with three parts:
BrickWright itself is a thin harness around a single agent: it writes one build.py containing createbuild() and its own runtests(), then loops over validate (fixed physical checks plus the agent's tests), render (multi-view sheet), and finalize. The system prompt carries only the BrickLayer docs and the loop instructions, with no worked examples or building tips, and permissions are narrower than the baselines: Codex and Claude Code baselines can read, write, and execute arbitrary files with the full LDraw docs and part library, while BrickWright agents can only edit build.py and run text-output Python snippets. Raw LDraw falls short because the format records absolute poses only, so mating two connectors means computing coordinates by hand; in the ablation, forcing raw LDraw roughly doubles token cost and triples validate calls.
Evaluated on BrickNet-512 (512 text prompts, VQA-captioned from human-made builds), all agents at High reasoning:
| Setup | Physical validity | Cost per prompt |
| Codex, GPT-6 Luna | 8.6% | $0.02 |
| BrickWright, GPT-6 Luna | 100% | $0.03 |
| Codex, GPT-6 Astra | 69.3% | $1.21 |
| BrickWright, GPT-6 Astra | 100% | $1.28 |
| BrickNet (specialized model) | 31.8% | — |
| Articraft (adapted), Astra | 69.3% | $3.00 |
The ablations split the two components. Without the validator (agents still told to stay valid, free to script their own checks), the best agent, Sol, drops to 77.9%, Haiku to 58.6%, Luna to 52.9%. Without the assembly API, forced into raw LDraw, Sol holds 100%, Luna 99.6%, Haiku 82.2%, but Luna's average part count collapses from 27.4 to 7.2 and the builds turn simplistic. The API alone, with no validator, already buys up to 44 points over the bare baseline (Luna: 8.6% to 52.9%).
Prompt adherence, scored by VQAScore, Perception Encoder, and SigLIP 2, rises for the lightweight models (Haiku 0.603 to 0.652 VQAScore, Luna 0.650 to 0.694) and holds flat for Sol and Astra. Most agents average 2 to 3 validate calls; Luna needs 4 to 5. A LEGO-experienced human re-checked flagged borderline cases, and the table reflects post-review numbers. Among specialized models, BrickGPT reaches 99% validity but averages 2.67 connected components and tracks prompts far worse, limited to 8 rectangular brick types over 21 categories. The framework also handles image-conditioned generation, a 29,667-part city scene, articulated builds, script editing, and generation constrained to a 20-brick subset, shown qualitatively.
The transferable finding: in any domain whose constraints can be written as executable checks (collision, connectivity, inventory), an ergonomic API plus a validator in a repair loop substitutes for task-specific fine-tuning. It held across four capability tiers, Haiku through Astra. Lightweight models gain the most: Luna with the toolkit reaches 100% validity and better prompt adherence at $0.03 per prompt, about 2% of the Astra setup. Output is standard LDraw over real parts, directly buildable. The cost is speed, 143 to 415 seconds per build against roughly 5 seconds for BrickNet, so throughput-critical uses still favor specialized models. The recipe itself is familiar; what is new is turning connector rules into deterministic checks, quantifying the result to 100%, and showing the API alone contributes roughly half the gain.
Stated by the authors: physical feasibility here excludes assembly-order executability and force-based stability, so a connected, collision-free build can still collapse mid-assembly. Agentic pipelines are slower and costlier than specialized models, and distilling these capabilities into small, fast models is left to future work.
Remaining concerns from the read: the referee for the 100% figure is BrickLayer's own validator, with human review only on auto-flagged cases, so generator and judge share one formalization. Prompt adherence relies on embedding- and VQA-style similarity, which is lenient on structural detail. The 512 prompts are VLM captions of human-made builds, skewing toward familiar objects, and the headline extensions (29k parts, articulation) appear only in qualitative figures. Astra was dropped from the ablations over budget.