Octrees as an Explicit 3D Language
Ran Dan, Si-Tong Wei, Pengfei Xiong, Wei Zhang, Yadong Mu, Peng-Shuai Wang
cs.CV
2026-10-02
OctLLM encodes shapes as sparse-octree tokens and adds full-rank 3D branches to a frozen Qwen2.5-VL: top multimodal LLM on 3D generation and captioning, MMLU unchanged.
Existing 3D LLMs pay on two fronts. On representation, ShapeLLM-Omni compresses shapes into VQVAE codebook indices, so fine geometry is gone before the model ever sees it; LLaMA-Mesh serializes vertex coordinates as plain text, burying locality and hierarchy in the character stream. On adaptation, these systems absorb the new modality by fine-tuning the backbone, and pretrained language behavior degrades with it. After 3D training, ShapeLLM-Omni drops from 64.65 to 30.17 on MMLU and loses 40.7 points on HellaSwag; LLaMA-Mesh falls to 34.03 on HellaSwag from the backbone-level 67.44.
OctLLM, from a Peking University team, treats these as two separate problems with two fixes: geometry enters as an explicit octree sequence, and 3D capacity lives in parameters fully disjoint from the pretrained weights.
Representation: the S-Octree. An octree recursively splits space into eight octants and subdivides only near surfaces. Serialization follows a Z-order traversal, so nodes adjacent in 3D tend to stay adjacent in the sequence, and every 8 binary occupancy bits pack into one byte-valued token in [0, 255], cutting sequence length 8x versus raw bits. Even so, a depth-6 tree (finest grid 64³) gets long, and the finest levels are highly redundant. OctLLM randomly empties half of the second-to-last-level nodes and drops their finest descendants, calling the result an S-Octree. A separately trained 3D U-Net restores the pruned fine occupancy, and an off-the-shelf TRELLIS decoder turns the completed grid into a textured mesh. The LLM only models global layout and part structure.
Architecture: token-routed dual streams. The Qwen2.5-VL-7B backbone stays frozen. In 12 of its 28 blocks, mesh tokens route through independent full-rank 3D branches covering the FFN and all four attention projections, while text and image tokens keep the frozen pathway; the two streams share one causal self-attention, so only token-local transforms are separated. The mesh vocabulary (256 byte tokens plus boundaries and <MASK>) gets its own embeddings and output head, so occupancy prediction never competes with word logits. Each octree token's three RoPE channels carry its node's 3D center coordinates, rescaled per level into a shared frame, plus a learnable depth embedding marking scale. During generation, a <MASK> query inserted before each byte carries the target position and depth, so the model only answers what occupancy belongs at that location; historical mask queries are suppressed as attention keys. Trainable parameters total about 2.80B, roughly a quarter of the model and a third of what full fine-tuning would touch.
| Setting | Metric | OctLLM | Comparison |
| Image-to-3D (Toys4K) | FID (Inc.) ↓ | 31.58 | ShapeLLM-Omni 38.21, a 17.4% cut |
| Image-to-3D | KID (Inc.) ↓ | 0.56 | ShapeLLM-Omni 1.02, down 45% |
| Text-to-3D | FID (Inc.) ↓ | 45.61 | best among multimodal LLMs; dedicated TRELLIS 30.90 |
| PointLLM-200 captioning | GPT-img ↑ | 46.88 | ShapeLLM-Omni 18.14 (+28.7); PointLLM-7B 45.63 |
| Language retention | MMLU / HellaSwag | 64.65 / 67.44 | identical to backbone; ShapeLLM-Omni 30.17 / 26.74 |
The ablations pin down each claim. Swapping the S-Octree for 1,024 VQVAE tokens with everything else fixed worsens image-to-3D FID from 26.31 to 33.10 and lowers both understanding metrics, so explicit coordinate-anchored tokens beat opaque latent indices. Replacing the full-rank branches with rank-256 token-routed LoRA preserves language just as well but drops FID to 32.44, and high-rank LoRA reportedly hits gradient explosions when trained at scale. Removing the <MASK> query hurts most, with FID rising to 61.08. Sparsification itself saves resources: peak memory falls from 58 GB to 44 GB and epoch time from 2.17 to 1.37 hours. At inference, 200 prompts average 1,277 output tokens and 37.6 seconds on an RTX 5090, slower than ShapeLLM-Omni's 19.9 seconds but far ahead of LLaMA-Mesh's 85.3.
The recipe, route a new modality through its own parameters, share attention, freeze the backbone, is directly reusable by any team extending an LLM to a new modality. Language preservation here is architectural, not a data-balancing act: a text-only conversation reproduces the backbone's computation exactly. The representation finding generalizes too, since letting the LLM read an explicit sequence anchored to coordinates and depth helps generation and understanding at once, with both directions sharing one 3D language.
Its position should be stated plainly: TRELLIS still generates better. OctLLM's pitch is one model that generates, describes, and chats without degrading. And since ShapeLLM-Omni and AR3D-R1 also use TRELLIS as their mesh decoder, OctLLM's lead over them comes from the representation and pipeline, not the decoder.
The S-Octree carries geometry only, no color or material, so appearance is undescribable; PointLLM reads colored point clouds and still leads on GPT-ref and semantic similarity, and PointLLM-13B's appendix GPT-img (48.06) edges out OctLLM's 46.88. The finest resolution is 64³, coarse for fine structure. IFEval drops from 75.41 to 69.87 because the mesh head occasionally opens a 3D sequence mid-answer; the fix is masking the mesh vocabulary at inference, which obviously cannot stay on when generating. All ablations run at ShapeNet-airplane scale, so placement and rank conclusions at full scale rest on indirect evidence. Understanding is evaluated on PointLLM-200 alone.