TaoTex: Boosting Texture Detail Fidelity for Native 3D Material Generation
Xiuchao Wu, Shuichang Lai, Jiangjing Lyu, Chengfei Lyu
cs.CV
2026-09-28
TaoTex paints PBR materials on a fixed mesh using 50K synthetic assets. Single-view albedo PSNR reaches 25.15 versus TRELLIS.2 at 21.92; LPIPS drops from 0.096 to 0.049.
Image-to-3D models can hold a shape together. The print on that shape often cannot. Cans, boxes, and garments have few corners, so geometry never says where a logo or a line of type should sit. Letters break up, or they drift onto the wrong face.
View-space methods borrow a 2D diffusion prior, paint several views, and bake them onto the mesh. Overlaps ghost. Occluded regions need a UV fill. Depth edges throw projection errors. Native 3D methods generate appearance in a space tied to the surface, so coverage stays consistent and self-occlusion is ordinary. TRELLIS packs shape and color into one structured latent, SLAT. NaTex splits structure from surface color. Smooth faces still come out wrong. Public 3D sets carry little high-frequency print, image features lean semantic, and a 3D VAE caps pixel accuracy. Multi-view input is either missing or falls apart once the views have to agree.
TaoTex narrows the job. Geometry is given. One to four images go in. Physically based channels come out: albedo, roughness, and metallic.
The backbone is the o-voxel latent from TRELLIS.2, a sparse voxel code aligned to space. A geometry encoder supplies the structural latent, a diffusion Transformer denoises with v-prediction, and a material decoder emits the PBR mesh.
An agent builds the high-frequency data. Qwen-3.8 writes Blender scripts for cans, cups, and boxes, inspects the render, revises the code, and unwraps UVs once the mesh passes. Qwen-Image-3.0-Pro paints text and patterns inside the UV mask. A second check scores sharpness and boundary fit. Approved maps receive roughness and metallic values. The pool is called HFT. Shape variety stays small on purpose: a fixed family of simple meshes gets many textures, because the shortage is print, not topology. Painted maps that miss the UV layout still go into training. HFT is 50K assets, plus 500K filtered Objaverse-XL assets and 300K from TexVerse. Each asset is rendered from 16 views at 1024, with random field of view and lighting.
Deep DINOv2 features wash out small type. MLFF reads layers 4, 11, 17, and 23, projects each, concatenates channels, reweights them with a channel gate, compresses the stack, and adds the mean residual across layers. Learnable patch position codes keep the fused vector anchored on the image.
Optimization runs in two phases of 100K steps. The first regresses v in latent space. The second decodes the prediction to a mesh, renders the three material channels, and applies L1, SSIM, and LPIPS. Views that match the input use weights 1.0, 0.2, and 0.2. Two extra novel views use 0.2, 0.2, and 1.0, so perceptual score outweighs a pixel lock. The material VAE is fine-tuned on 16 H20 GPUs and then frozen. The DiT trains at batch 16 and learning rate 1e-4, alternating single-view and multi-view batches.
Concatenating multi-view tokens with no view label fragments the print and sticks it on the wrong face. A learnable embedding marks front, back, left, and right, spanning 90 degrees horizontally and 180 degrees in elevation, up to four images, and stays fixed at inference. Training is rich in cylinders and boxes, whose outlines barely change with viewpoint, so the model cannot read the camera off the shape and ignore the embedding. Timesteps come from a logit-normal with mean 2 and standard deviation 1, biased toward low SNR, because cross-view placement is settled in the early steps, close to pure noise.
The quantitative set is 30 richly textured objects from TexVerse. Every method except ReconViaGen receives ground-truth geometry. ReconViaGen is qualitative only, because it cannot take a mesh.
Single-view albedo on the conditioning view:
| Method | PSNR↑ | SSIM↑ | LPIPS↓ |
| MaterialMVP | 18.90 | 0.830 | 0.160 |
| UniTEX | 21.84 | 0.873 | 0.101 |
| Ink3D | 19.39 | 0.837 | 0.155 |
| TRELLIS.2 | 21.92 | 0.879 | 0.096 |
| TaoTex | 25.15 | 0.914 | 0.049 |
Shaded renders under the same protocol reach 26.52 PSNR for TaoTex, 23.46 for TRELLIS.2, and 22.81 for UniTEX. Novel-view albedo FID is 96.90, against 113.86 for TRELLIS.2 and 143.89 for Ink3D. Shaded-render FID is 77.66 against 87.69. CLIP image similarity barely moves: 0.9096 versus 0.9063 on novel-view albedo.
With four input views, albedo PSNR is 25.48 for TaoTex, 20.18 for MaterialMVP, and 19.91 for TRELLIS.2. TRELLIS.2 falls below its own single-view score of 21.92. MaterialMVP rises from 18.90 to 20.18 and still trails by about 5 dB. TRELLIS.2 swaps the conditioning image mid-denoising and often prints text on the wrong side of a cylinder. ReconViaGen, driven by VGGT features for both shape and texture, lands marks near the right place and then blurs them.
The four-view ablation on albedo PSNR reads 22.90 without HFT, 23.40 without MLFF, 24.47 without viewpoint embeddings, 24.61 without the pixel-space loss, and 25.48 for the full model. Drop HFT and the text is illegible. Train only in latent space and the tiny characters warp once you zoom in.
Rodin Gen2.5, Tripo v3.1, and Hunyuan v3.1 appear only in qualitative comparisons. TaoTex is given Rodin's mesh. The commercial systems use their own. On symmetric bottles and vases they ghost and misalign. Different geometry means this is not a leaderboard.
Production meshes often exist before anyone has a readable package label, a fabric print, or separated PBR channels. One photo is enough to run. Four photos pin symmetric objects down. Against TRELLIS.2, single-view albedo PSNR is about 3.2 dB higher, and LPIPS falls from 0.096 to 0.049. At four views the gap exceeds 5 dB, and TRELLIS.2 scores worse than it did from one image.
The task is still materials on a mesh someone else built. The benchmark feeds ground-truth geometry. The gain is large inside that box.
Voxel resolution still bounds the smallest text and patterns. The paper says a higher resolution does not fit in GPU memory.
The 30 test objects come from TexVerse, and training already includes 300K TexVerse assets. Nothing says the 30 were held out. Overlap would inflate PSNR. HFT meshes are simple cans and boxes that unwrap cleanly. Folded garments are untested. Viewpoint embeddings cover only four canonical cameras. The low-SNR schedule, the symmetric shapes, and the embeddings are trained as one bundle, and the ablation removes only the embeddings.
Novel-view CLIP-I beats TRELLIS.2 by about 0.003. When the check that matters is object identity, pixel error is where TaoTex pulls away, and the table looks larger than that check.