Does Native 3D Texture Generation Necessarily Require 3D Assets for Training?
Jiangshan Wang, Zeqiang Lai, Jiayi Guo, Xin Yang, Xin Huang, Jiarui Chen, Ziheng Ouyang, Chunchao Guo, Xiangyu Yue
cs.CV
2026-09-28
Tex-Zero turns 2D images into 3D training data via patch-wise rotation, trains VAE and DiT on 11.1M images with zero 3D assets, and hits 35.31 PSNR vs 27.49 for 3D-trained NaTex.
Native 3D texture generation predicts a color for every point on a given 3D surface, conditioned on a set of multi-view reference images. The older pipeline generates 2D views first, then projects, fuses, and bakes them onto the geometry, and each stage compounds projection error and cross-view inconsistency. Going native removes those errors but creates a data problem: training has historically required large corpora of textured 3D assets, which are expensive to produce and scarce next to 2D images.
The paper starts from a structural observation. A 2D image and a 3D texture are the same kind of data: both assign colors to spatial positions, and the only difference is whether the positions live on a plane or on a curved surface. If that holds, the endless supply of high-quality images might convert directly into training samples for 3D texture generation.
Tex-Zero says yes, and trains both the VAE (the autoencoder that compresses colors into a latent space) and the DiT (a diffusion Transformer) on images alone. The working hypothesis: what training samples must supply is fine-grained, high-quality color; geometry matters less and can be built by hand rather than sourced from semantically meaningful assets.
The data pipeline has four steps:
The Tex-Zero VAE swaps the dense 2D operations of the FLUX VAE for sparse 3D operations. A warm-up phase is mandatory: rotate the whole plane without cutting it first; run the full pipeline from scratch and the VAE never converges. The loss combines pointwise color reconstruction, a perceptual term (invert the patch transforms, reassemble the original 2D image, compute LPIPS against the source), and KL regularization. The useful side effect: this VAE reconstructs 2D images presented as 3D planes with high fidelity, while 3D-asset-trained VAEs blur them out. Reference images and target textures can therefore share one encoder and one latent space, closing the representation gap at conditioning time.
The Tex-Zero DiT uses the FLUX MMDiT backbone with a standard flow-matching objective (the network regresses a velocity field from noise toward data). The six canonical reference views are placed as planes oriented by viewing direction, encoded by the frozen Tex-Zero VAE, and concatenated into the conditioning sequence; geometry enters as surface normals concatenated along channels. Positional encoding is a four-dimensional RoPE: one axis tags whether a token belongs to the target texture or one of the six views, three axes carry latent-grid coordinates. Each training step resamples the number and combination of conditioning views, and random condition dropout enables classifier-free guidance. Because patches in the constructed samples occlude each other, the model must infer hidden regions from visible context, which acts as natural outpainting training and maps directly onto occluded surfaces of real assets at inference.
Training data: SA-1B, BLIP3o-60k, and ShareGPT-4o, about 11.1M images total, resized to 1536x1536. Zero 3D assets anywhere.
VAE reconstruction at the f16c16 configuration (the one the DiT uses):
| Model | Training data | 3D recon LPIPS↓ | PSNR-PC↑ | Rendered PSNR↑ | 2D recon LPIPS↓ |
| Same-architecture Sparse VAE | 1M 3D assets | 0.0345 | 30.90 | 38.97 | 0.2993 |
| Tex-Zero VAE | images only | 0.0316 | 32.59 | 38.54 | 0.1792 |
A VAE that never saw a 3D asset reconstructs real assets with lower LPIPS and higher PSNR-PC, slightly lower rendered PSNR, and clearly better 2D image reconstruction. The generation gap is much wider:
| Method | Front-view LPIPS↓ | PSNR↑ | SSIM↑ | Six-view LPIPS↓ | PSNR↑ |
| NaTex | 0.0669 | 27.49 | 0.947 | 0.0754 | 27.74 |
| TRELLIS.2 | 0.1187 | 21.38 | 0.900 | – | – |
| Tex-Zero | 0.0294 | 35.31 | 0.985 | 0.0340 | 35.68 |
Under front-view conditioning, PSNR sits 7.82 above NaTex and LPIPS is less than half of it.
Three ablations each carry a lesson. Every layer of data construction helps: a fixed plane gives 0.2160 LPIPS, random rotation of the whole plane 0.1107, four rotated patches 0.0591, adding aggregation 0.0513, sixteen patches with aggregation 0.0355. Conditioning decides detail fidelity: DINO encoding 0.1228, a separate image VAE 0.0838, a shared 3D-trained VAE 0.0835, and the image-trained unified VAE best at 0.0481. When 3D assets do exist, mixing in 2D images still helps: VAE LPIPS drops from 0.0343 to 0.0140 and DiT LPIPS from 0.0302 to 0.0216 versus 3D-only training.
Data scarcity is what blocks scaling in 3D generation. If color-geometry pairs can be constructed artificially, the data supply for texture models switches from collecting 3D assets to collecting images, and internet-scale 2D data becomes directly usable. The joint-training result shows this is not a fallback for teams without 3D data; teams holding assets gain from adding images anyway.
The unified 2D-3D latent space also fixes a longstanding conditioning problem: encoding reference images with an image encoder or a 3D VAE loses fine-grained detail before the DiT ever sees it. Synthetic-data training already generalized in 3D reconstruction (LRM-Zero, MegaSynth, VFusion3D); this paper pushes the idea into native texture generation. For 3D teams it is a data-strategy-level result, and the code is open-sourced.
The paper has no dedicated limitations section; the items below are partly inferred from the text rather than author-stated. Constructed samples contain no semantically meaningful object geometry, so the model never sees a coherent whole object during training, and generalization to real assets rests on coverage of local orientations and occlusion patterns. Why that coverage suffices is analyzed in an appendix; the main text offers only ablation evidence. The same-architecture control trains on about one million in-house 3D assets versus 11.1M images, an order-of-magnitude data gap, so how much of the generation-side lead comes from the paradigm versus sheer data volume cannot be separated. Whether NaTex and TRELLIS.2 were trained under comparable compute is not stated, and TRELLIS.2 lacks six-view numbers. VAE reconstruction is not a clean sweep either: at f8c16, PSNR is 42.41 against 44.71 for the 3D-trained counterpart, with LPIPS being the metric Tex-Zero wins. Ablations run for only 20K steps, so their ordering may not hold at full training length. Evaluation is reference-based against ground-truth renders, with dataset details in the appendix. The task boundary also matters: geometry plus multi-view references in, surface colors out, with no text-driven texturing and no material or lighting semantics.