JoVox: Joint Hierarchical Flow Matching for Lightweight Voxel Generation
Xiangjun Tang, Jianqi Chen, Peter Wonka
cs.CV, cs.GT
2026-11-07
JoVox jointly models all voxel-hierarchy levels with discrete flow matching so finer cells inform coarse ones, beating TRELLIS.2 with 8x training throughput and 11% lower Chamfer.
Voxels record which cells of a 3D grid a shape occupies. They are the lightweight intermediate of choice for 3D pipelines: TRELLIS-style models generate a 64³ voxel grid first and refine it into a mesh. Both existing routes to high-fidelity voxels carry a structural cost.
Latent methods (TRELLIS, TRELLIS.2) compress voxels with an autoencoder and generate the latent grid. Fidelity is high, but there is an extra model to train, and the generator still processes every latent cell, empty space included, so cost grows with the cube of resolution. Hierarchical methods (XCube, OctFusion, OctGPT) refine a 4³ coarse grid level by level up to 64³ and touch only occupied cells, which keeps them cheap. Their flaw is that the coarse levels, which decide where every finer cell may appear, are generated when the least is known about the shape and never revisited. The paper calls them "plausible but wrong", and the finer levels inherit the errors.
JoVox replaces per-level factorization with the joint distribution over the whole hierarchy, so finer cells take part in deciding coarse ones. Three components:
The transformer matches the TRELLIS.2 voxel-stage architecture: all levels share one sequence with full self-attention, and image conditions enter through cross-attention on DINOv3 features. During training, mask ratios are drawn independently per level, so any intermediate state seen at inference lies in the training distribution. The loss is averaged within each level before averaging across levels, so the numerous fine levels do not dominate, and low unmask ratios (where predictions are least certain) are sampled more often.
Image-conditioned generation: trained on 174,528 assets from TRELLIS-500K, tested on 1,000 unseen Toys4k objects.
| Method | CDV ↓ | IoU ↑ | LPIPSV ↓ | CDshell (×10⁻³) ↓ |
| TRELLIS.2 | 0.01250 | 0.4017 | 0.1524 | 0.524 |
| JoVox | 0.01107 | 0.4158 | 0.1395 | 0.407 |
That is 11% lower voxel Chamfer, and after the same shared mesh stage, 22% lower shell Chamfer; JoVox ranks first on all eight voxel and mesh metrics. On unconditional ShapeNet generation (chair, airplane, table, rifle, averaged), it wins all six metrics: FIDV 2.06 versus 10.1 for the best baseline (80% lower), FIDVD 15.3 (64% lower), and 1-NNA 0.574, closest to the ideal 0.5.
The efficiency gap is larger. Same backbone, one 80 GB A100, batch size 32: peak memory 29.0 GB versus 47.3 GB, 8x training throughput, and 0.82 versus 12.2 TFLOPs per forward pass (about 1/15), because a training state averages 273 token cells while TRELLIS.2 processes all 4,096 cells of its 16³ latent grid for every shape. Full training used 4 H100s: after roughly two days (80K steps) it surpassed TRELLIS (trained for 400K steps) on every metric, and after about two and a half days it matched TRELLIS.2, which was trained on 32 H100s.
Ablations separate the contributions: single-level prediction reaches 0.02150 CDV, two-level 0.01941, and adding joint modeling 0.01506, so joint modeling contributes the most. Against the MDLM, ReMDM, ReMDM-conf, and MaskGIT samplers, the remasking strategy gives the best CDV and IoU.
For 3D generation practitioners, this is a route that beats the latent SOTA in quality at an order of magnitude lower cost, and full training fits on 4 GPUs, within reach of academic groups. For generative modeling more broadly, letting early decisions be revised by later information is not specific to voxels; any level-wise generator faces the same coarse-level lock-in, and joint modeling plus remasking is a transferable fix. Honest framing: an 11% Chamfer gain is incremental; the substance here is the efficiency side.
The authors state two: the training set is about 175K assets, under a quarter of TRELLIS.2's roughly 800K, so more data likely helps; and codes unmasked in the same step are still sampled independently, which can leave them inconsistent, with remasking a mitigation rather than a fix. Two more caveats from this read: matching TRELLIS.2 "in two and a half days" leans on a step count TRELLIS.2 never disclosed, so it is an approximate alignment; and the inference-speed comparison and 256³ results appear only as appendix references, with no numbers in the main text.